2024-03-30T11:48:09.981 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T11:48:09.982 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612 --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} -- /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/orig.config.yaml 2024-03-30T11:48:10.030 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T11:48:10.118 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612 branch: wip-ceph_test_rados-partial-reads description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} email: yweinste@redhat.com first_in_suite: false job_id: '7630612' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-ceph_test_rados-partial-reads ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd beacon report interval: 30 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 1 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - must scrub before tier agent can activate - \(POOL_APP_NOT_ENABLED\) sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ceph_test_rados-partial-reads sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9805 sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ceph_test_rados-partial-reads suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPFboCyO3bM9cLXOTIqjPA3FxZk5ahQ8MGtbCVb0ySQ79yoypJc3uYDzYtn5h4bDP1IlOWFsHQn5j7M2CKKkts= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPP32OwpGxEByXBvXQdTzC8LSFKogGaZO56cXXA/etDCnz6VFskXcEmmXaqaKJT5/21SSm5rA5Ikq1zBoq5ODBw= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - ceph config set mgr mgr/balancer/log_level debug - ceph balancer status - ceph osd set-require-min-compat-client reef - ceph balancer mode upmap-read - ceph balancer on - ceph balancer status - ceph balancer status detail - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 map_discontinuity_sleep_time: 200 timeout: 1800 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache min_read_recency_for_promote 0 - sudo ceph osd pool set cache min_write_recency_for_promote 0 - rados: clients: - client.0 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-30_00:41:04 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T11:48:10.118 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa; will attempt to use it 2024-03-30T11:48:10.119 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks 2024-03-30T11:48:10.119 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T11:48:10.121 INFO:teuthology.task.internal:Checking packages... 2024-03-30T11:48:10.145 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc' 2024-03-30T11:48:10.145 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T11:48:10.145 INFO:teuthology.packaging:ref: None 2024-03-30T11:48:10.146 INFO:teuthology.packaging:tag: None 2024-03-30T11:48:10.146 INFO:teuthology.packaging:branch: wip-ceph_test_rados-partial-reads 2024-03-30T11:48:10.146 INFO:teuthology.packaging:sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T11:48:10.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-ceph_test_rados-partial-reads 2024-03-30T11:48:10.286 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T11:48:10.288 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T11:48:10.294 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T11:48:10.294 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T11:48:10.327 INFO:teuthology.task.internal:Saving configuration 2024-03-30T11:48:10.345 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T11:48:10.364 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T11:48:10.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 11:42:06.869316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN7l0cynW1MHN2ZN8JD4rMW7UlSoxIIy4k31bEcNmU63'} 2024-03-30T11:48:10.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 11:42:06.870627', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC27EztkF5D47DZ7d5tzkf8SSmsgY1oWySiYMg5K2tgL'} 2024-03-30T11:48:10.411 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T11:48:10.448 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-30T11:48:10.448 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-30T11:48:10.449 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T11:48:10.525 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3f8f61a430>, signals=[15]) 2024-03-30T11:48:10.525 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T11:48:10.531 INFO:teuthology.task.internal:Opening connections... 2024-03-30T11:48:10.531 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T11:48:10.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:48:10.613 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T11:48:10.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:48:10.691 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T11:48:10.697 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-30T11:48:10.701 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-30T11:48:10.702 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-30T11:48:10.749 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-30T11:48:10.749 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="22.04" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=jammy 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-30T11:48:10.750 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=jammy 2024-03-30T11:48:10.751 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-30T11:48:10.773 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-03-30T11:48:10.778 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-03-30T11:48:10.779 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-03-30T11:48:10.824 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-30T11:48:10.824 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Ubuntu" 2024-03-30T11:48:10.824 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="22.04" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_CODENAME=jammy 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:ID=ubuntu 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE=debian 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-30T11:48:10.825 INFO:teuthology.orchestra.run.smithi178.stdout:UBUNTU_CODENAME=jammy 2024-03-30T11:48:10.825 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-03-30T11:48:10.848 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T11:48:10.859 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T11:48:10.945 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T11:48:10.946 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-30T11:48:10.949 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-03-30T11:48:10.954 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T11:48:10.983 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T11:48:10.983 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-30T11:48:10.995 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-03-30T11:48:11.005 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T11:48:11.062 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T11:48:11.077 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T11:48:11.077 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T11:48:11.078 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T11:48:11.078 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T11:48:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T11:48:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T11:48:11.086 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-30T11:48:11.086 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-30T11:48:11.133 INFO:teuthology.orchestra.run.smithi178.stdout:5.15.0-56-generic 2024-03-30T11:48:11.133 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.15.0-56-generic 2024-03-30T11:48:11.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-03-30T11:48:11.134 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-56-generic 2024-03-30T11:48:11.134 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.15.0-56-generic 2024-03-30T11:48:11.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-03-30T11:48:11.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-30T11:48:11.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-03-30T11:48:11.455 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:48:11.457 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-30T11:48:11.458 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T11:48:11.539 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-30T11:48:11.577 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:48:11.579 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T11:48:11.634 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-30T11:48:11.696 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-03-30T11:48:11.769 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-03-30T11:48:11.809 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-03-30T11:48:11.814 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-03-30T11:48:11.817 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-30T11:48:11.817 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-03-30T11:48:11.841 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-30T11:48:11.841 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-03-30T11:48:11.845 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T11:48:11.845 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-30T11:48:11.853 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-30T11:48:11.859 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-30T11:48:11.860 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-30T11:48:11.860 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-30T11:48:11.885 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-30T11:48:11.885 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-30T11:48:11.885 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-30T11:48:11.885 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-30T11:48:11.889 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-30T11:48:11.891 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-30T11:48:11.891 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-30T11:48:11.892 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-30T11:48:11.910 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-30T11:48:11.912 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T11:48:11.912 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-30T11:48:11.918 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-30T11:48:11.927 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-30T11:48:11.929 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-30T11:48:11.934 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-30T11:48:11.934 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-30T11:48:11.935 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-30T11:48:11.935 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-30T11:48:11.935 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-30T11:48:11.936 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-30T11:48:11.955 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-30T11:48:11.955 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-30T11:48:11.955 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-30T11:48:11.955 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-30T11:48:11.975 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-30T11:48:11.975 INFO:teuthology.orchestra.run.smithi178.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-30T11:48:11.993 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-30T11:48:11.998 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-03-30T11:48:12.128 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-03-30T11:48:12.416 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-03-30T11:48:12.433 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-30T11:48:12.434 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-03-30T11:48:12.444 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-30T11:48:12.444 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-03-30T11:48:12.472 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-30T11:48:12.487 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-30T11:48:12.488 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-30T11:48:12.489 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-30T11:48:12.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-30T11:48:12.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-03-30T11:48:12.558 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-30T11:48:12.558 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T11:48:12.558 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-30T11:48:12.563 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T11:48:12.578 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-30T11:48:12.728 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-30T11:48:12.758 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-30T11:48:13.166 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-30T11:48:13.171 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-30T11:48:13.186 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-30T11:48:13.186 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-30T11:48:13.188 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-30T11:48:13.306 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-30T11:48:13.417 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-30T11:48:13.437 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-30T11:48:13.439 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-30T11:48:13.443 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-30T11:48:13.443 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-30T11:48:13.444 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-30T11:48:13.444 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-30T11:48:13.448 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-30T11:48:13.452 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-30T11:48:13.618 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-30T11:48:13.701 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-30T11:48:13.782 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-30T11:48:13.863 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-30T11:48:13.893 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-30T11:48:17.536 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 12.6 MB in 2s (6,949 kB/s) 2024-03-30T11:48:18.154 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 12.6 MB in 3s (4,975 kB/s) 2024-03-30T11:48:18.934 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:48:18.952 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T11:48:19.010 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:48:19.205 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-03-30T11:48:19.205 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-03-30T11:48:19.372 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:48:19.372 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:48:19.373 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:48:19.373 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:48:19.373 INFO:teuthology.orchestra.run.smithi178.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-30T11:48:19.373 INFO:teuthology.orchestra.run.smithi178.stdout: python2.7-minimal 2024-03-30T11:48:19.373 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:48:19.374 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-03-30T11:48:19.374 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T11:48:19.374 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-30T11:48:19.375 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-30T11:48:19.376 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-03-30T11:48:19.376 INFO:teuthology.orchestra.run.smithi178.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-30T11:48:19.376 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-03-30T11:48:19.376 INFO:teuthology.orchestra.run.smithi178.stdout: thermald 2024-03-30T11:48:19.426 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-03-30T11:48:19.426 INFO:teuthology.orchestra.run.smithi178.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T11:48:19.426 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-30T11:48:19.427 INFO:teuthology.orchestra.run.smithi178.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-30T11:48:19.427 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages will be upgraded: 2024-03-30T11:48:19.428 INFO:teuthology.orchestra.run.smithi178.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-30T11:48:19.486 INFO:teuthology.orchestra.run.smithi178.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T11:48:19.486 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 113 MB of archives. 2024-03-30T11:48:19.486 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-30T11:48:19.486 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-30T11:48:19.494 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:48:19.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T11:48:19.573 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:48:19.776 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-30T11:48:19.777 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-30T11:48:19.949 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:48:19.950 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:48:19.950 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:48:19.951 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:48:19.951 INFO:teuthology.orchestra.run.smithi172.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-30T11:48:19.951 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7-minimal 2024-03-30T11:48:19.951 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:48:19.951 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-03-30T11:48:19.952 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T11:48:19.952 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-30T11:48:19.952 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-30T11:48:19.953 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-03-30T11:48:19.953 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-30T11:48:19.954 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-03-30T11:48:19.954 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2024-03-30T11:48:19.981 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-30T11:48:19.994 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-03-30T11:48:19.994 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T11:48:19.994 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-30T11:48:19.994 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-30T11:48:19.996 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-03-30T11:48:19.996 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-30T11:48:20.059 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T11:48:20.060 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 113 MB of archives. 2024-03-30T11:48:20.060 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-30T11:48:20.060 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-30T11:48:20.085 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-30T11:48:20.761 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-30T11:48:20.966 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-30T11:48:21.287 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-30T11:48:21.287 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-30T11:48:21.288 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-30T11:48:21.690 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-30T11:48:21.778 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-30T11:48:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-30T11:48:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-30T11:48:22.003 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-30T11:48:22.207 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 113 MB in 2s (48.3 MB/s) 2024-03-30T11:48:22.238 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-30T11:48:22.296 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-30T11:48:22.412 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-30T11:48:22.698 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 113 MB in 2s (49.4 MB/s) 2024-03-30T11:48:22.894 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-30T11:48:24.485 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-30T11:48:24.489 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:24.692 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:25.051 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-30T11:48:25.055 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:25.271 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:27.193 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-30T11:48:27.214 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:27.313 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:27.447 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-30T11:48:27.468 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:27.577 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:27.763 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-30T11:48:27.783 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:27.821 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:28.083 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-30T11:48:28.104 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:28.141 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:34.714 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:34.830 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:35.172 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:35.324 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:35.446 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:35.575 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:35.658 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:35.754 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:35.878 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-30T11:48:35.901 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-30T11:48:35.963 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T11:48:36.082 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-30T11:48:36.104 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-30T11:48:36.149 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T11:48:41.330 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-30T11:48:41.367 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:41.405 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:42.341 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-30T11:48:42.379 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T11:48:42.540 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:44.186 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:44.294 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:44.661 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T11:48:44.783 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:44.909 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-30T11:48:45.071 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:45.291 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T11:48:45.442 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T11:48:45.808 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T11:48:45.914 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:46.056 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-30T11:48:46.185 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:46.389 INFO:teuthology.orchestra.run.smithi178.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:48:46.389 INFO:teuthology.orchestra.run.smithi178.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-30T11:48:46.389 INFO:teuthology.orchestra.run.smithi178.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:48:46.389 INFO:teuthology.orchestra.run.smithi178.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-30T11:48:46.558 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:47.507 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:48:47.507 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-30T11:48:47.507 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:48:47.507 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-30T11:48:47.754 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:48.092 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:49.163 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:49.519 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-30T11:48:49.645 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-30T11:48:49.746 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:49.803 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-30T11:48:49.803 INFO:teuthology.orchestra.run.smithi178.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:48:50.514 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-30T11:48:50.673 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-30T11:48:50.799 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T11:48:50.869 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-30T11:48:50.870 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:49:04.305 INFO:teuthology.orchestra.run.smithi178.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-30T11:49:04.305 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub' 2024-03-30T11:49:04.321 INFO:teuthology.orchestra.run.smithi178.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T11:49:04.352 INFO:teuthology.orchestra.run.smithi178.stdout:Generating grub configuration file ... 2024-03-30T11:49:04.717 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:49:04.736 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:49:05.070 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-30T11:49:05.070 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2024-03-30T11:49:05.093 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:49:05.099 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T11:49:05.105 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T11:49:05.132 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2024-03-30T11:49:05.175 INFO:teuthology.orchestra.run.smithi178.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T11:49:05.180 INFO:teuthology.orchestra.run.smithi178.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T11:49:05.337 INFO:teuthology.orchestra.run.smithi178.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T11:49:05.338 INFO:teuthology.orchestra.run.smithi178.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T11:49:05.338 INFO:teuthology.orchestra.run.smithi178.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T11:49:05.379 INFO:teuthology.orchestra.run.smithi178.stdout:done 2024-03-30T11:49:05.439 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:49:05.458 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:49:05.770 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:49:05.775 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T11:49:05.848 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T11:49:05.854 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T11:49:05.861 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 21 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-03-30T11:49:05.885 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.15.0.101.98 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-03-30T11:49:05.886 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-03-30T11:49:05.886 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-30T11:49:05.886 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T11:49:05.886 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T11:49:05.887 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T11:49:05.887 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T11:49:05.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get clean 2024-03-30T11:49:06.014 INFO:teuthology.orchestra.run.smithi172.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T11:49:06.014 INFO:teuthology.orchestra.run.smithi172.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T11:49:06.015 INFO:teuthology.orchestra.run.smithi172.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T11:49:06.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-03-30T11:49:06.066 INFO:teuthology.orchestra.run.smithi172.stdout:done 2024-03-30T11:49:06.206 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T11:49:06.297 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:49:06.378 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T11:49:06.466 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T11:49:06.584 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-03-30T11:49:06.601 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.101.98 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-03-30T11:49:06.602 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-03-30T11:49:06.602 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-30T11:49:06.602 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T11:49:06.602 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T11:49:06.602 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T11:49:06.603 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T11:49:06.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-03-30T11:49:06.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-30T11:49:06.922 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:49:06.923 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T11:49:06.935 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T11:49:06.954 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T11:49:07.561 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:49:07.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T11:49:07.638 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:49:07.829 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-03-30T11:49:07.830 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-03-30T11:49:07.999 INFO:teuthology.orchestra.run.smithi178.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-30T11:49:07.999 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:49:07.999 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:49:07.999 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:49:07.999 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:49:08.000 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T11:49:08.000 INFO:teuthology.orchestra.run.smithi178.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T11:49:08.000 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:49:08.043 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T11:49:08.045 DEBUG:teuthology.orchestra.run.smithi178:> dpkg -s linux-image-generic 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Package: linux-image-generic 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Status: install ok installed 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Priority: optional 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Section: kernel 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Installed-Size: 21 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: amd64 2024-03-30T11:49:08.058 INFO:teuthology.orchestra.run.smithi178.stdout:Source: linux-meta 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout:Version: 5.15.0.101.98 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout:Recommends: thermald 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout:Description: Generic Linux kernel image 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout: This package will always depend on the latest generic kernel image 2024-03-30T11:49:08.059 INFO:teuthology.orchestra.run.smithi178.stdout: available. 2024-03-30T11:49:08.059 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-03-30T11:49:08.104 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.uR7mdGhzwJ 2024-03-30T11:49:08.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uR7mdGhzwJ 2024-03-30T11:49:08.160 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.uR7mdGhzwJ 2024-03-30T11:49:08.276 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:49:08.286 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.uR7mdGhzwJ is 10KB 2024-03-30T11:49:08.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T11:49:08.300 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.uR7mdGhzwJ 2024-03-30T11:49:08.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-30T11:49:08.346 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:49:08.359 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T11:49:08.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-30T11:49:08.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-30T11:49:08.423 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-30T11:49:08.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo update-grub 2024-03-30T11:49:08.529 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-30T11:49:08.529 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-30T11:49:08.683 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-30T11:49:08.683 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T11:49:08.684 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:49:08.721 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T11:49:08.723 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-03-30T11:49:08.780 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.101.98 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-03-30T11:49:08.781 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-03-30T11:49:08.781 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-30T11:49:08.824 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.oS9k5wmUH5 2024-03-30T11:49:08.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oS9k5wmUH5 2024-03-30T11:49:08.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.oS9k5wmUH5 2024-03-30T11:49:08.986 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.oS9k5wmUH5 is 10KB 2024-03-30T11:49:08.999 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.oS9k5wmUH5 2024-03-30T11:49:09.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-30T11:49:09.058 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T11:49:09.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-30T11:49:09.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-30T11:49:09.145 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-30T11:49:09.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2024-03-30T11:49:09.751 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub' 2024-03-30T11:49:09.753 INFO:teuthology.orchestra.run.smithi178.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T11:49:09.763 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-03-30T11:49:09.944 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:49:09.950 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:49:10.207 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:49:10.212 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T11:49:10.255 INFO:teuthology.orchestra.run.smithi178.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T11:49:10.260 INFO:teuthology.orchestra.run.smithi178.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T11:49:10.377 INFO:teuthology.orchestra.run.smithi178.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T11:49:10.377 INFO:teuthology.orchestra.run.smithi178.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T11:49:10.377 INFO:teuthology.orchestra.run.smithi178.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T11:49:10.407 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-03-30T11:49:10.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-03-30T11:49:10.438 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2024-03-30T11:49:10.439 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T11:49:10.445 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-30T11:49:10.653 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T11:49:10.658 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T11:49:10.896 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T11:49:10.901 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T11:49:10.942 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T11:49:10.947 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T11:49:11.062 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T11:49:11.062 INFO:teuthology.orchestra.run.smithi172.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T11:49:11.062 INFO:teuthology.orchestra.run.smithi172.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T11:49:11.093 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-30T11:49:11.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-30T11:49:40.442 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T11:49:40.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:49:40.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:49:41.105 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T11:49:41.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:49:41.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:49:58.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-30T11:49:59.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-30T11:50:07.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:50:07.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:50:08.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:50:08.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:50:11.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-30T11:50:11.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-30T11:50:23.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:50:23.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:50:23.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:50:23.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:50:26.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-30T11:50:41.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:50:41.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:51:12.981 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-30T11:51:13.593 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:51:13.593 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-30T11:51:13.593 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-30T11:51:13.640 INFO:teuthology.orchestra.run.smithi178.stdout:5.15.0-101-generic 2024-03-30T11:51:13.640 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-30T11:51:13.640 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T11:51:13.640 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-30T11:51:14.641 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T11:51:14.641 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T11:51:14.714 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-03-30T11:51:14.737 DEBUG:teuthology.parallel:result is None 2024-03-30T11:51:23.523 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T11:51:38.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:51:38.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:51:38.770 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-30T11:51:39.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:51:39.407 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-30T11:51:39.407 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-30T11:51:39.412 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-101-generic 2024-03-30T11:51:39.412 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-30T11:51:39.412 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T11:51:39.412 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-30T11:51:40.413 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T11:51:40.413 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T11:51:40.497 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-30T11:51:40.528 DEBUG:teuthology.parallel:result is None 2024-03-30T11:51:40.529 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T11:51:40.540 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T11:51:40.540 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T11:51:40.543 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T11:51:40.549 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T11:51:40.555 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T11:51:40.565 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T11:51:40.565 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T11:51:40.591 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T11:51:40.618 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T11:51:40.624 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T11:51:40.624 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T11:51:40.643 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T11:51:40.661 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:51:40.670 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:51:40.676 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:51:40.683 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:51:40.685 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T11:51:40.692 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T11:51:40.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T11:51:40.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T11:51:40.741 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T11:51:40.751 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T11:51:40.751 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T11:51:40.771 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T11:51:40.788 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T11:51:40.822 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T11:51:40.870 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T11:51:40.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T11:51:40.929 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T11:51:40.935 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T11:51:40.980 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T11:51:40.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T11:51:41.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-30T11:51:41.068 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-03-30T11:51:41.139 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T11:51:41.147 INFO:teuthology.task.internal:Starting timer... 2024-03-30T11:51:41.147 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T11:51:41.161 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T11:51:41.169 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2024-03-30T11:51:41.169 DEBUG:teuthology.task.selinux:Excluding smithi178: OS 'ubuntu' does not support SELinux 2024-03-30T11:51:41.169 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T11:51:41.169 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-30T11:51:41.169 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T11:51:41.169 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T11:51:41.179 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T11:51:41.197 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T11:51:41.198 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T11:58:41.230 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-03-30T11:58:41.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:58:41.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:58:41.319 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-30T11:58:41.398 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-30T11:58:41.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:58:41.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T11:58:41.489 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-30T11:58:41.563 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-30T11:58:41.563 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T11:58:41.574 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T11:58:41.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T11:58:41.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T11:58:41.578 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T11:58:41.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T11:58:41.601 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-30T11:58:41.601 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Command line: ntpd -gq 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: ---------------------------------------------------- 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: ntp-4 is maintained by Network Time Foundation, 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: corporation. Support and training for ntp-4 are 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: available at https://www.nwtime.org/support 2024-03-30T11:58:41.602 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: ---------------------------------------------------- 2024-03-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: proto: precision = 0.045 usec (-24) 2024-03-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: basedate set to 2022-02-04 2024-03-30T11:58:41.603 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: gps base set to 2022-02-06 (week 2196) 2024-03-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T11:58:41.604 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: restrict ::: KOD does nothing without LIMITED. 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen and drop on 0 v6wildcard [::]:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen normally on 2 lo 127.0.0.1:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen normally on 4 lo [::1]:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2024-03-30T11:58:41.606 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:41 ntpd[17191]: Listening on routing socket on fd #22 for interface updates 2024-03-30T11:58:41.634 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-30T11:58:41.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Command line: ntpd -gq 2024-03-30T11:58:41.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: ---------------------------------------------------- 2024-03-30T11:58:41.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: ntp-4 is maintained by Network Time Foundation, 2024-03-30T11:58:41.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-30T11:58:41.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: corporation. Support and training for ntp-4 are 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: available at https://www.nwtime.org/support 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: ---------------------------------------------------- 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: proto: precision = 0.045 usec (-24) 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: basedate set to 2022-02-04 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: gps base set to 2022-02-06 (week 2196) 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T11:58:41.636 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T11:58:41.637 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: restrict ::: KOD does nothing without LIMITED. 2024-03-30T11:58:41.637 INFO:teuthology.orchestra.run.smithi178.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-30T11:58:41.638 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen and drop on 0 v6wildcard [::]:123 2024-03-30T11:58:41.638 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-30T11:58:41.638 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen normally on 2 lo 127.0.0.1:123 2024-03-30T11:58:41.638 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen normally on 3 enp3s0f1 172.21.15.178:123 2024-03-30T11:58:41.638 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen normally on 4 lo [::1]:123 2024-03-30T11:58:41.639 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ef9%5]:123 2024-03-30T11:58:41.639 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:41 ntpd[17170]: Listening on routing socket on fd #22 for interface updates 2024-03-30T11:58:42.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:42.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:42.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:42.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:42 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:44.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:44 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:44.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:44 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:45.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:45 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:45.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:45.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:45.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:45.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:45.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:45 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:46.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:46 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:46.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:46 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:47.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:47.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:47 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:47.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:47 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:48.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:48.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:48.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:48.602 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stdout:30 Mar 11:58:48 ntpd[17191]: ntpd: time slew -0.001383 s 2024-03-30T11:58:48.603 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.001383s 2024-03-30T11:58:48.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-30T11:58:48.604 INFO:teuthology.orchestra.run.smithi172.stderr:30 Mar 11:58:48 ntpd[17191]: can't open /var/log/ntpstats/loopstats.20240330: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:48.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:48 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:48.662 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T11:58:48.662 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-03-30T11:58:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:48.663 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:49.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:49.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:49.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:49.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:49.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:49.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:49 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T11:58:50.635 INFO:teuthology.orchestra.run.smithi178.stdout:30 Mar 11:58:50 ntpd[17170]: ntpd: time slew -0.001320 s 2024-03-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi178.stdout:ntpd: time slew -0.001320s 2024-03-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-30T11:58:50.636 INFO:teuthology.orchestra.run.smithi178.stderr:30 Mar 11:58:50 ntpd[17170]: can't open /var/log/ntpstats/loopstats.20240330: Permission denied 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:50.683 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T11:58:50.683 INFO:teuthology.run_tasks:Running task install... 2024-03-30T11:58:50.692 DEBUG:teuthology.task.install:project ceph 2024-03-30T11:58:50.692 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}} 2024-03-30T11:58:50.692 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'} 2024-03-30T11:58:50.692 INFO:teuthology.task.install:Using flavor: default 2024-03-30T11:58:50.698 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T11:58:50.698 INFO:teuthology.task.install:extra packages: [] 2024-03-30T11:58:50.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2024-03-30T11:58:50.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-key list | grep Ceph 2024-03-30T11:58:50.763 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-30T11:58:50.778 INFO:teuthology.orchestra.run.smithi178.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-30T11:58:50.790 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-30T11:58:50.790 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2024-03-30T11:58:50.791 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-30T11:58:50.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T11:58:50.806 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-30T11:58:50.807 INFO:teuthology.orchestra.run.smithi178.stdout:uid [ unknown] Ceph.com (release key) 2024-03-30T11:58:50.807 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-30T11:58:50.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T11:58:51.011 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default/ 2024-03-30T11:58:51.012 INFO:teuthology.task.install.deb:Package version is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T11:58:51.020 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default/ 2024-03-30T11:58:51.021 INFO:teuthology.task.install.deb:Package version is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T11:58:51.125 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T11:58:51.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-30T11:58:51.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-30T11:58:51.191 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T11:58:51.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-30T11:58:51.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo apt-get update 2024-03-30T11:58:51.319 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T11:58:51.381 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:58:51.381 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T11:58:51.394 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T11:58:51.415 INFO:teuthology.orchestra.run.smithi178.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T11:58:51.420 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy InRelease 2024-03-30T11:58:51.426 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T11:58:51.429 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy InRelease 2024-03-30T11:58:51.450 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-30T11:58:51.457 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-30T11:58:51.471 INFO:teuthology.orchestra.run.smithi178.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-30T11:58:51.493 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-30T11:58:51.493 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-30T11:58:51.510 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T11:58:51.519 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-30T11:58:51.530 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-30T11:58:51.572 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-30T11:58:51.589 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T11:58:51.731 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 28.9 kB in 0s (70.5 kB/s) 2024-03-30T11:58:51.740 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 28.9 kB in 0s (81.8 kB/s) 2024-03-30T11:58:52.680 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:58:52.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy cephadm=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-mgr=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph-fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy ceph-volume=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python3-rados=19.0.0-1864-g5dc2ea30-1jammy python3-rgw=19.0.0-1864-g5dc2ea30-1jammy python3-cephfs=19.0.0-1864-g5dc2ea30-1jammy python3-rbd=19.0.0-1864-g5dc2ea30-1jammy libcephfs2=19.0.0-1864-g5dc2ea30-1jammy libcephfs-dev=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T11:58:52.710 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:58:52.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy cephadm=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-mgr=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph-fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy ceph-volume=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python3-rados=19.0.0-1864-g5dc2ea30-1jammy python3-rgw=19.0.0-1864-g5dc2ea30-1jammy python3-cephfs=19.0.0-1864-g5dc2ea30-1jammy python3-rbd=19.0.0-1864-g5dc2ea30-1jammy libcephfs2=19.0.0-1864-g5dc2ea30-1jammy libcephfs-dev=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T11:58:52.758 INFO:teuthology.orchestra.run.smithi178.stdout:Reading package lists... 2024-03-30T11:58:52.786 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-30T11:58:52.954 INFO:teuthology.orchestra.run.smithi178.stdout:Building dependency tree... 2024-03-30T11:58:52.955 INFO:teuthology.orchestra.run.smithi178.stdout:Reading state information... 2024-03-30T11:58:52.981 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-30T11:58:52.982 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-30T11:58:53.144 INFO:teuthology.orchestra.run.smithi178.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:58:53.145 INFO:teuthology.orchestra.run.smithi178.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:58:53.145 INFO:teuthology.orchestra.run.smithi178.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:58:53.145 INFO:teuthology.orchestra.run.smithi178.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:58:53.146 INFO:teuthology.orchestra.run.smithi178.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T11:58:53.146 INFO:teuthology.orchestra.run.smithi178.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T11:58:53.146 INFO:teuthology.orchestra.run.smithi178.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:58:53.146 INFO:teuthology.orchestra.run.smithi178.stdout:The following additional packages will be installed: 2024-03-30T11:58:53.146 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-30T11:58:53.148 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T11:58:53.149 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet zip 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout:Suggested packages: 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-30T11:58:53.150 INFO:teuthology.orchestra.run.smithi178.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-30T11:58:53.151 INFO:teuthology.orchestra.run.smithi178.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-30T11:58:53.151 INFO:teuthology.orchestra.run.smithi178.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-30T11:58:53.151 INFO:teuthology.orchestra.run.smithi178.stdout:Recommended packages: 2024-03-30T11:58:53.151 INFO:teuthology.orchestra.run.smithi178.stdout: btrfs-tools 2024-03-30T11:58:53.173 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T11:58:53.173 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T11:58:53.173 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T11:58:53.173 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T11:58:53.174 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T11:58:53.174 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T11:58:53.174 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T11:58:53.176 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-30T11:58:53.177 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet zip 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-03-30T11:58:53.178 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout:The following NEW packages will be installed: 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-30T11:58:53.251 INFO:teuthology.orchestra.run.smithi178.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T11:58:53.253 INFO:teuthology.orchestra.run.smithi178.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-30T11:58:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-30T11:58:53.277 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-30T11:58:53.278 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T11:58:53.279 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-30T11:58:53.280 INFO:teuthology.orchestra.run.smithi172.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-30T11:58:53.394 INFO:teuthology.orchestra.run.smithi178.stdout:0 upgraded, 118 newly installed, 0 to remove and 298 not upgraded. 2024-03-30T11:58:53.395 INFO:teuthology.orchestra.run.smithi178.stdout:Need to get 176 MB of archives. 2024-03-30T11:58:53.395 INFO:teuthology.orchestra.run.smithi178.stdout:After this operation, 730 MB of additional disk space will be used. 2024-03-30T11:58:53.395 INFO:teuthology.orchestra.run.smithi178.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,545 kB] 2024-03-30T11:58:53.439 INFO:teuthology.orchestra.run.smithi178.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-30T11:58:53.467 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 118 newly installed, 0 to remove and 298 not upgraded. 2024-03-30T11:58:53.467 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 176 MB of archives. 2024-03-30T11:58:53.467 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 730 MB of additional disk space will be used. 2024-03-30T11:58:53.467 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-30T11:58:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,545 kB] 2024-03-30T11:58:53.611 INFO:teuthology.orchestra.run.smithi178.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,220 kB] 2024-03-30T11:58:53.717 INFO:teuthology.orchestra.run.smithi178.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1864-g5dc2ea30-1jammy [709 kB] 2024-03-30T11:58:53.740 INFO:teuthology.orchestra.run.smithi178.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1864-g5dc2ea30-1jammy [356 kB] 2024-03-30T11:58:53.755 INFO:teuthology.orchestra.run.smithi178.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1864-g5dc2ea30-1jammy [32.8 kB] 2024-03-30T11:58:53.756 INFO:teuthology.orchestra.run.smithi178.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1864-g5dc2ea30-1jammy [180 kB] 2024-03-30T11:58:53.763 INFO:teuthology.orchestra.run.smithi178.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1864-g5dc2ea30-1jammy [64.6 kB] 2024-03-30T11:58:53.765 INFO:teuthology.orchestra.run.smithi178.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1864-g5dc2ea30-1jammy [330 kB] 2024-03-30T11:58:53.776 INFO:teuthology.orchestra.run.smithi178.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1864-g5dc2ea30-1jammy [6,575 kB] 2024-03-30T11:58:53.864 INFO:teuthology.orchestra.run.smithi178.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-30T11:58:53.875 INFO:teuthology.orchestra.run.smithi178.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-30T11:58:53.933 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-30T11:58:53.945 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-30T11:58:53.957 INFO:teuthology.orchestra.run.smithi178.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-30T11:58:54.015 INFO:teuthology.orchestra.run.smithi178.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1864-g5dc2ea30-1jammy [112 kB] 2024-03-30T11:58:54.017 INFO:teuthology.orchestra.run.smithi178.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1864-g5dc2ea30-1jammy [466 kB] 2024-03-30T11:58:54.026 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-30T11:58:54.032 INFO:teuthology.orchestra.run.smithi178.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1864-g5dc2ea30-1jammy [24.2 MB] 2024-03-30T11:58:54.187 INFO:teuthology.orchestra.run.smithi178.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-30T11:58:54.198 INFO:teuthology.orchestra.run.smithi178.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-30T11:58:54.228 INFO:teuthology.orchestra.run.smithi178.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-30T11:58:54.237 INFO:teuthology.orchestra.run.smithi178.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-30T11:58:54.238 INFO:teuthology.orchestra.run.smithi178.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-30T11:58:54.240 INFO:teuthology.orchestra.run.smithi178.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-30T11:58:54.256 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-30T11:58:54.258 INFO:teuthology.orchestra.run.smithi178.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-30T11:58:54.267 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-30T11:58:54.297 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-30T11:58:54.306 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-30T11:58:54.308 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-30T11:58:54.309 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-30T11:58:54.327 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-30T11:58:54.426 INFO:teuthology.orchestra.run.smithi178.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-30T11:58:54.496 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-30T11:58:54.836 INFO:teuthology.orchestra.run.smithi178.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-30T11:58:54.852 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,220 kB] 2024-03-30T11:58:54.869 INFO:teuthology.orchestra.run.smithi178.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-30T11:58:54.871 INFO:teuthology.orchestra.run.smithi178.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-30T11:58:54.902 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-30T11:58:54.936 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-30T11:58:54.938 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-30T11:58:54.985 INFO:teuthology.orchestra.run.smithi178.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-30T11:58:54.987 INFO:teuthology.orchestra.run.smithi178.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1864-g5dc2ea30-1jammy [5,081 kB] 2024-03-30T11:58:55.002 INFO:teuthology.orchestra.run.smithi178.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-30T11:58:55.018 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-30T11:58:55.024 INFO:teuthology.orchestra.run.smithi178.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-30T11:58:55.031 INFO:teuthology.orchestra.run.smithi178.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-30T11:58:55.032 INFO:teuthology.orchestra.run.smithi178.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-30T11:58:55.043 INFO:teuthology.orchestra.run.smithi178.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-30T11:58:55.044 INFO:teuthology.orchestra.run.smithi178.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-30T11:58:55.070 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-30T11:58:55.091 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-30T11:58:55.098 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-30T11:58:55.099 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-30T11:58:55.100 INFO:teuthology.orchestra.run.smithi178.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1864-g5dc2ea30-1jammy [243 kB] 2024-03-30T11:58:55.108 INFO:teuthology.orchestra.run.smithi178.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [124 kB] 2024-03-30T11:58:55.110 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-30T11:58:55.111 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-30T11:58:55.113 INFO:teuthology.orchestra.run.smithi178.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1864-g5dc2ea30-1jammy [1,521 kB] 2024-03-30T11:58:55.169 INFO:teuthology.orchestra.run.smithi178.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1864-g5dc2ea30-1jammy [6,247 kB] 2024-03-30T11:58:55.210 INFO:teuthology.orchestra.run.smithi178.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-30T11:58:55.277 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-30T11:58:55.293 INFO:teuthology.orchestra.run.smithi178.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-30T11:58:55.321 INFO:teuthology.orchestra.run.smithi178.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-30T11:58:55.338 INFO:teuthology.orchestra.run.smithi178.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-30T11:58:55.361 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-30T11:58:55.383 INFO:teuthology.orchestra.run.smithi178.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-30T11:58:55.389 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-30T11:58:55.402 INFO:teuthology.orchestra.run.smithi178.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1864-g5dc2ea30-1jammy [22.8 MB] 2024-03-30T11:58:55.407 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-30T11:58:55.452 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-30T11:58:55.598 INFO:teuthology.orchestra.run.smithi178.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-30T11:58:55.613 INFO:teuthology.orchestra.run.smithi178.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-30T11:58:55.619 INFO:teuthology.orchestra.run.smithi178.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-30T11:58:55.625 INFO:teuthology.orchestra.run.smithi178.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-30T11:58:55.646 INFO:teuthology.orchestra.run.smithi178.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-30T11:58:55.650 INFO:teuthology.orchestra.run.smithi178.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-30T11:58:55.664 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-30T11:58:55.679 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-30T11:58:55.686 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-30T11:58:55.691 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-30T11:58:55.713 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-30T11:58:55.716 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-30T11:58:55.798 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1864-g5dc2ea30-1jammy [709 kB] 2024-03-30T11:58:55.816 INFO:teuthology.orchestra.run.smithi178.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-30T11:58:55.882 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-30T11:58:56.020 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1864-g5dc2ea30-1jammy [356 kB] 2024-03-30T11:58:56.123 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1864-g5dc2ea30-1jammy [32.8 kB] 2024-03-30T11:58:56.133 INFO:teuthology.orchestra.run.smithi178.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-30T11:58:56.134 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1864-g5dc2ea30-1jammy [180 kB] 2024-03-30T11:58:56.172 INFO:teuthology.orchestra.run.smithi178.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-30T11:58:56.184 INFO:teuthology.orchestra.run.smithi178.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-30T11:58:56.188 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1864-g5dc2ea30-1jammy [64.6 kB] 2024-03-30T11:58:56.199 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-30T11:58:56.209 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1864-g5dc2ea30-1jammy [330 kB] 2024-03-30T11:58:56.238 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-30T11:58:56.249 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-30T11:58:56.308 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1864-g5dc2ea30-1jammy [6,575 kB] 2024-03-30T11:58:56.317 INFO:teuthology.orchestra.run.smithi178.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [14.0 kB] 2024-03-30T11:58:56.317 INFO:teuthology.orchestra.run.smithi178.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [867 kB] 2024-03-30T11:58:56.334 INFO:teuthology.orchestra.run.smithi178.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1864-g5dc2ea30-1jammy [2,190 kB] 2024-03-30T11:58:56.372 INFO:teuthology.orchestra.run.smithi178.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1864-g5dc2ea30-1jammy [777 kB] 2024-03-30T11:58:56.381 INFO:teuthology.orchestra.run.smithi178.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1864-g5dc2ea30-1jammy [144 kB] 2024-03-30T11:58:56.383 INFO:teuthology.orchestra.run.smithi178.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1864-g5dc2ea30-1jammy [1,923 kB] 2024-03-30T11:58:56.389 INFO:teuthology.orchestra.run.smithi178.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-30T11:58:56.390 INFO:teuthology.orchestra.run.smithi178.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-30T11:58:56.396 INFO:teuthology.orchestra.run.smithi178.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-30T11:58:56.401 INFO:teuthology.orchestra.run.smithi178.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-30T11:58:56.417 INFO:teuthology.orchestra.run.smithi178.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-30T11:58:56.438 INFO:teuthology.orchestra.run.smithi178.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-30T11:58:56.444 INFO:teuthology.orchestra.run.smithi178.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-30T11:58:56.454 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-30T11:58:56.455 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-30T11:58:56.456 INFO:teuthology.orchestra.run.smithi178.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1864-g5dc2ea30-1jammy [8,625 kB] 2024-03-30T11:58:56.461 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-30T11:58:56.467 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-30T11:58:56.482 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-30T11:58:56.503 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-30T11:58:56.509 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-30T11:58:56.619 INFO:teuthology.orchestra.run.smithi178.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-30T11:58:56.684 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-30T11:58:56.776 INFO:teuthology.orchestra.run.smithi178.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1864-g5dc2ea30-1jammy [14.2 kB] 2024-03-30T11:58:56.776 INFO:teuthology.orchestra.run.smithi178.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1864-g5dc2ea30-1jammy [49.9 MB] 2024-03-30T11:58:56.872 INFO:teuthology.orchestra.run.smithi178.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-30T11:58:56.950 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-30T11:58:57.049 INFO:teuthology.orchestra.run.smithi178.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-30T11:58:57.063 INFO:teuthology.orchestra.run.smithi178.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-30T11:58:57.082 INFO:teuthology.orchestra.run.smithi178.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-30T11:58:57.128 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-30T11:58:57.142 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-30T11:58:57.147 INFO:teuthology.orchestra.run.smithi178.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-30T11:58:57.162 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-30T11:58:57.226 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-30T11:58:57.437 INFO:teuthology.orchestra.run.smithi178.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-30T11:58:57.439 INFO:teuthology.orchestra.run.smithi178.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-30T11:58:57.440 INFO:teuthology.orchestra.run.smithi178.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-30T11:58:57.441 INFO:teuthology.orchestra.run.smithi178.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-30T11:58:57.443 INFO:teuthology.orchestra.run.smithi178.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-30T11:58:57.515 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-30T11:58:57.517 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-30T11:58:57.521 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-30T11:58:57.522 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-30T11:58:57.524 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-30T11:58:57.667 INFO:teuthology.orchestra.run.smithi178.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-30T11:58:57.747 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-30T11:58:58.102 INFO:teuthology.orchestra.run.smithi178.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-30T11:58:58.108 INFO:teuthology.orchestra.run.smithi178.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-30T11:58:58.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-30T11:58:58.191 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-30T11:58:58.298 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1864-g5dc2ea30-1jammy [112 kB] 2024-03-30T11:58:58.334 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1864-g5dc2ea30-1jammy [466 kB] 2024-03-30T11:58:58.398 INFO:teuthology.orchestra.run.smithi178.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-30T11:58:58.399 INFO:teuthology.orchestra.run.smithi178.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-30T11:58:58.401 INFO:teuthology.orchestra.run.smithi178.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-30T11:58:58.406 INFO:teuthology.orchestra.run.smithi178.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-30T11:58:58.407 INFO:teuthology.orchestra.run.smithi178.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-30T11:58:58.409 INFO:teuthology.orchestra.run.smithi178.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-30T11:58:58.410 INFO:teuthology.orchestra.run.smithi178.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-30T11:58:58.411 INFO:teuthology.orchestra.run.smithi178.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-30T11:58:58.479 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1864-g5dc2ea30-1jammy [24.2 MB] 2024-03-30T11:58:58.481 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-30T11:58:58.482 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-30T11:58:58.484 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-30T11:58:58.486 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-30T11:58:58.487 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-30T11:58:58.490 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-30T11:58:58.491 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-30T11:58:58.493 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-30T11:58:58.590 INFO:teuthology.orchestra.run.smithi178.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-30T11:58:58.674 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-30T11:58:58.816 INFO:teuthology.orchestra.run.smithi178.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1864-g5dc2ea30-1jammy [121 kB] 2024-03-30T11:58:58.818 INFO:teuthology.orchestra.run.smithi178.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1864-g5dc2ea30-1jammy [38.4 kB] 2024-03-30T11:58:58.820 INFO:teuthology.orchestra.run.smithi178.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1864-g5dc2ea30-1jammy [13.0 MB] 2024-03-30T11:58:59.003 INFO:teuthology.orchestra.run.smithi178.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-30T11:58:59.049 INFO:teuthology.orchestra.run.smithi178.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-30T11:58:59.084 INFO:teuthology.orchestra.run.smithi178.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-30T11:58:59.086 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-30T11:58:59.166 INFO:teuthology.orchestra.run.smithi178.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [91.7 kB] 2024-03-30T11:58:59.183 INFO:teuthology.orchestra.run.smithi178.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-30T11:58:59.192 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-30T11:58:59.194 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-30T11:58:59.223 INFO:teuthology.orchestra.run.smithi178.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-30T11:58:59.235 INFO:teuthology.orchestra.run.smithi178.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-30T11:58:59.241 INFO:teuthology.orchestra.run.smithi178.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-30T11:58:59.287 INFO:teuthology.orchestra.run.smithi178.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-30T11:58:59.290 INFO:teuthology.orchestra.run.smithi178.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-30T11:58:59.311 INFO:teuthology.orchestra.run.smithi178.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-30T11:58:59.353 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-30T11:58:59.359 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-30T11:58:59.361 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-30T11:58:59.363 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-30T11:58:59.386 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-30T11:58:59.390 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-30T11:58:59.413 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-30T11:58:59.476 INFO:teuthology.orchestra.run.smithi178.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-30T11:58:59.577 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-30T11:58:59.750 INFO:teuthology.orchestra.run.smithi178.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-30T11:58:59.767 INFO:teuthology.orchestra.run.smithi178.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-30T11:58:59.850 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-30T11:58:59.867 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-30T11:58:59.922 INFO:teuthology.orchestra.run.smithi178.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-30T11:58:59.952 INFO:teuthology.orchestra.run.smithi178.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-30T11:59:00.020 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-30T11:59:00.049 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-30T11:59:00.109 INFO:teuthology.orchestra.run.smithi178.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-30T11:59:00.111 INFO:teuthology.orchestra.run.smithi178.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-30T11:59:00.114 INFO:teuthology.orchestra.run.smithi178.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-30T11:59:00.136 INFO:teuthology.orchestra.run.smithi178.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-30T11:59:00.141 INFO:teuthology.orchestra.run.smithi178.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-30T11:59:00.206 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-30T11:59:00.209 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-30T11:59:00.210 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-30T11:59:00.233 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-30T11:59:00.238 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-30T11:59:00.815 INFO:teuthology.orchestra.run.smithi178.stdout:Fetched 176 MB in 7s (25.2 MB/s) 2024-03-30T11:59:00.976 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-30T11:59:01.076 INFO:teuthology.orchestra.run.smithi178.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-30T11:59:01.082 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-30T11:59:01.174 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T11:59:01.486 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-30T11:59:01.513 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-30T11:59:01.544 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T11:59:01.847 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-30T11:59:01.873 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-30T11:59:01.914 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T11:59:02.346 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1864-g5dc2ea30-1jammy [5,081 kB] 2024-03-30T11:59:02.392 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-30T11:59:02.419 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:02.467 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:02.879 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-30T11:59:02.905 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:02.945 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:02.959 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1864-g5dc2ea30-1jammy [243 kB] 2024-03-30T11:59:02.986 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [124 kB] 2024-03-30T11:59:02.996 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1864-g5dc2ea30-1jammy [1,521 kB] 2024-03-30T11:59:03.213 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1864-g5dc2ea30-1jammy [6,247 kB] 2024-03-30T11:59:03.240 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-30T11:59:03.266 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:03.298 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:03.668 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-30T11:59:03.694 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-30T11:59:03.734 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T11:59:03.934 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1864-g5dc2ea30-1jammy [22.8 MB] 2024-03-30T11:59:04.013 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librados2. 2024-03-30T11:59:04.028 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../007-librados2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:04.062 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:04.515 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librbd1. 2024-03-30T11:59:04.541 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../008-librbd1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:04.582 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:05.210 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs2. 2024-03-30T11:59:05.236 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:05.276 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:05.562 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rados. 2024-03-30T11:59:05.588 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:05.629 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:05.948 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-30T11:59:05.974 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:06.015 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:06.250 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cephfs. 2024-03-30T11:59:06.265 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:06.308 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:06.464 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [14.0 kB] 2024-03-30T11:59:06.465 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [867 kB] 2024-03-30T11:59:06.502 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1864-g5dc2ea30-1jammy [2,190 kB] 2024-03-30T11:59:06.628 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-30T11:59:06.654 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:06.685 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1864-g5dc2ea30-1jammy [777 kB] 2024-03-30T11:59:06.694 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:06.753 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1864-g5dc2ea30-1jammy [144 kB] 2024-03-30T11:59:06.758 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1864-g5dc2ea30-1jammy [1,923 kB] 2024-03-30T11:59:06.929 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1864-g5dc2ea30-1jammy [8,625 kB] 2024-03-30T11:59:07.014 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-30T11:59:07.040 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-30T11:59:07.081 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T11:59:07.350 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-prettytable. 2024-03-30T11:59:07.377 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-30T11:59:07.408 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-30T11:59:07.588 INFO:teuthology.orchestra.run.smithi172.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1864-g5dc2ea30-1jammy [14.2 kB] 2024-03-30T11:59:07.588 INFO:teuthology.orchestra.run.smithi172.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1864-g5dc2ea30-1jammy [49.9 MB] 2024-03-30T11:59:07.661 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rbd. 2024-03-30T11:59:07.687 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:07.727 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:08.072 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-30T11:59:08.098 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-30T11:59:08.139 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T11:59:08.408 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-30T11:59:08.423 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-30T11:59:08.458 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T11:59:08.752 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-30T11:59:08.767 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-30T11:59:08.810 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T11:59:09.147 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua5.1. 2024-03-30T11:59:09.174 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-30T11:59:09.205 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-30T11:59:09.542 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-any. 2024-03-30T11:59:09.557 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-30T11:59:09.600 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-30T11:59:09.852 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package zip. 2024-03-30T11:59:09.868 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-30T11:59:09.911 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking zip (3.0-12build2) ... 2024-03-30T11:59:10.548 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package unzip. 2024-03-30T11:59:10.574 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-30T11:59:10.614 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-30T11:59:10.622 INFO:teuthology.orchestra.run.smithi172.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1864-g5dc2ea30-1jammy [121 kB] 2024-03-30T11:59:10.625 INFO:teuthology.orchestra.run.smithi172.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1864-g5dc2ea30-1jammy [38.4 kB] 2024-03-30T11:59:10.626 INFO:teuthology.orchestra.run.smithi172.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1864-g5dc2ea30-1jammy [13.0 MB] 2024-03-30T11:59:10.985 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package luarocks. 2024-03-30T11:59:11.011 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-30T11:59:11.020 INFO:teuthology.orchestra.run.smithi172.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [91.7 kB] 2024-03-30T11:59:11.051 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-30T11:59:11.426 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package librgw2. 2024-03-30T11:59:11.452 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../025-librgw2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:11.492 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:11.581 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 176 MB in 18s (9,929 kB/s) 2024-03-30T11:59:11.711 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-30T11:59:11.811 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-30T11:59:11.815 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-30T11:59:11.885 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T11:59:12.037 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rgw. 2024-03-30T11:59:12.052 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:12.094 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:12.197 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-30T11:59:12.223 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-30T11:59:12.263 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T11:59:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-30T11:59:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-30T11:59:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T11:59:12.616 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-30T11:59:12.642 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-30T11:59:12.683 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T11:59:12.775 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libradosstriper1. 2024-03-30T11:59:12.802 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:12.842 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:13.120 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-30T11:59:13.146 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:13.194 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:13.270 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-common. 2024-03-30T11:59:13.296 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:13.345 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:13.556 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-30T11:59:13.582 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:13.623 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:13.943 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-30T11:59:13.969 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T11:59:14.009 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T11:59:14.337 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-30T11:59:14.363 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-30T11:59:14.404 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T11:59:14.584 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-base. 2024-03-30T11:59:14.610 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:14.651 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:14.665 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librados2. 2024-03-30T11:59:14.692 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-librados2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:14.890 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:15.263 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-30T11:59:15.289 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-30T11:59:15.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librbd1. 2024-03-30T11:59:15.378 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-librbd1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:15.418 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:15.480 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-30T11:59:15.841 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cheroot. 2024-03-30T11:59:15.854 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2024-03-30T11:59:15.867 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-30T11:59:15.880 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:15.907 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T11:59:15.920 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:16.206 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2024-03-30T11:59:16.227 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-30T11:59:16.232 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:16.253 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-30T11:59:16.264 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:16.293 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-30T11:59:16.568 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-30T11:59:16.594 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:16.596 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-30T11:59:16.623 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-30T11:59:16.634 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:16.663 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-30T11:59:16.887 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2024-03-30T11:59:16.913 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:16.941 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-30T11:59:16.953 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:16.967 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-30T11:59:17.007 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-30T11:59:17.264 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-30T11:59:17.277 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-tempora. 2024-03-30T11:59:17.291 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:17.303 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-30T11:59:17.331 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:17.343 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-30T11:59:17.668 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-30T11:59:17.680 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-portend. 2024-03-30T11:59:17.694 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-30T11:59:17.706 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-30T11:59:17.734 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T11:59:17.746 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-30T11:59:18.029 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2024-03-30T11:59:18.032 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-30T11:59:18.055 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-30T11:59:18.059 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-30T11:59:18.095 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-30T11:59:18.099 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-30T11:59:18.314 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2024-03-30T11:59:18.341 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:18.381 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:18.385 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-30T11:59:18.412 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-30T11:59:18.443 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-30T11:59:18.726 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-30T11:59:18.752 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-30T11:59:18.780 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-natsort. 2024-03-30T11:59:18.794 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T11:59:18.807 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-30T11:59:18.847 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-30T11:59:19.053 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-30T11:59:19.069 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-30T11:59:19.111 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T11:59:19.133 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-logutils. 2024-03-30T11:59:19.149 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-30T11:59:19.191 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-30T11:59:19.406 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-30T11:59:19.421 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-30T11:59:19.444 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-30T11:59:19.456 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T11:59:19.471 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-30T11:59:19.511 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-30T11:59:19.760 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua5.1. 2024-03-30T11:59:19.786 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-30T11:59:19.822 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-mako. 2024-03-30T11:59:19.849 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-30T11:59:19.888 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T11:59:19.976 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-30T11:59:20.217 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-30T11:59:20.243 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-30T11:59:20.283 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-30T11:59:20.287 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-any. 2024-03-30T11:59:20.315 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-30T11:59:20.354 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-30T11:59:20.544 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-30T11:59:20.560 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-30T11:59:20.573 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package zip. 2024-03-30T11:59:20.589 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-30T11:59:20.594 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-30T11:59:20.623 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking zip (3.0-12build2) ... 2024-03-30T11:59:20.885 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package unzip. 2024-03-30T11:59:20.897 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webob. 2024-03-30T11:59:20.912 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-30T11:59:20.924 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-30T11:59:20.951 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-30T11:59:20.964 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-30T11:59:21.350 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-waitress. 2024-03-30T11:59:21.355 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package luarocks. 2024-03-30T11:59:21.377 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-30T11:59:21.382 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-30T11:59:21.419 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T11:59:21.422 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-30T11:59:21.728 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-tempita. 2024-03-30T11:59:21.754 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-30T11:59:21.794 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T11:59:21.805 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2024-03-30T11:59:21.831 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-librgw2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:21.871 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:22.081 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-paste. 2024-03-30T11:59:22.107 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-30T11:59:22.139 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T11:59:22.391 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2024-03-30T11:59:22.406 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:22.449 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-30T11:59:22.466 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-30T11:59:22.500 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T11:59:22.768 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-30T11:59:22.795 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-30T11:59:22.803 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-30T11:59:22.830 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-30T11:59:22.835 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T11:59:22.870 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-30T11:59:23.129 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2024-03-30T11:59:23.156 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:23.173 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-webtest. 2024-03-30T11:59:23.196 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:23.199 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-30T11:59:23.239 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-30T11:59:23.550 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pecan. 2024-03-30T11:59:23.577 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-30T11:59:23.583 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2024-03-30T11:59:23.609 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:23.617 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T11:59:23.674 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:23.963 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-certifi. 2024-03-30T11:59:23.990 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-30T11:59:24.030 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-30T11:59:24.300 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-idna. 2024-03-30T11:59:24.327 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-30T11:59:24.366 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-30T11:59:24.660 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-urllib3. 2024-03-30T11:59:24.687 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-30T11:59:24.727 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T11:59:24.930 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2024-03-30T11:59:24.956 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:25.005 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:25.089 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-requests. 2024-03-30T11:59:25.116 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-30T11:59:25.155 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T11:59:25.466 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-30T11:59:25.493 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-30T11:59:25.533 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T11:59:25.569 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-30T11:59:25.596 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-30T11:59:25.636 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-30T11:59:25.855 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-30T11:59:25.871 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:25.930 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cheroot. 2024-03-30T11:59:25.957 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-30T11:59:25.988 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T11:59:26.011 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:26.275 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-30T11:59:26.301 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-30T11:59:26.333 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-30T11:59:26.374 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-30T11:59:26.401 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:26.432 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:26.586 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-30T11:59:26.612 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-30T11:59:26.652 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-30T11:59:26.955 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-30T11:59:26.982 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-30T11:59:27.022 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-30T11:59:27.324 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempora. 2024-03-30T11:59:27.351 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-30T11:59:27.391 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-30T11:59:27.536 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-30T11:59:27.563 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-30T11:59:27.602 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-30T11:59:27.703 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-portend. 2024-03-30T11:59:27.730 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-30T11:59:27.761 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-30T11:59:27.905 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-openssl. 2024-03-30T11:59:27.932 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-30T11:59:27.971 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-30T11:59:28.047 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-30T11:59:28.074 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-30T11:59:28.113 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-30T11:59:28.258 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr. 2024-03-30T11:59:28.285 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:28.324 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:28.491 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-30T11:59:28.518 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-30T11:59:28.558 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-30T11:59:28.719 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mon. 2024-03-30T11:59:28.746 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:28.786 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:28.878 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2024-03-30T11:59:28.894 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-30T11:59:28.928 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-30T11:59:29.223 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2024-03-30T11:59:29.250 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-30T11:59:29.290 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-30T11:59:29.342 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-osd. 2024-03-30T11:59:29.368 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:29.400 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:29.592 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-30T11:59:29.619 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-30T11:59:29.659 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-30T11:59:29.929 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2024-03-30T11:59:29.946 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-30T11:59:29.987 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T11:59:30.473 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-30T11:59:30.478 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph. 2024-03-30T11:59:30.491 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-30T11:59:30.505 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../066-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:30.523 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-30T11:59:30.544 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:30.809 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-30T11:59:30.837 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-30T11:59:30.848 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-fuse. 2024-03-30T11:59:30.874 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:30.876 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-30T11:59:30.925 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:31.179 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2024-03-30T11:59:31.206 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-30T11:59:31.245 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-30T11:59:31.384 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mds. 2024-03-30T11:59:31.411 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:31.442 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:31.632 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2024-03-30T11:59:31.659 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-30T11:59:31.701 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T11:59:31.837 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package cephadm. 2024-03-30T11:59:31.864 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../069-cephadm_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:31.904 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:31.993 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2024-03-30T11:59:32.020 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-30T11:59:32.060 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T11:59:32.207 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-30T11:59:32.234 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-30T11:59:32.273 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-30T11:59:32.296 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2024-03-30T11:59:32.311 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-30T11:59:32.354 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T11:59:32.601 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-30T11:59:32.628 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T11:59:32.657 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-30T11:59:32.667 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T11:59:32.684 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-30T11:59:32.724 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T11:59:32.994 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-30T11:59:33.021 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-30T11:59:33.060 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-30T11:59:33.363 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2024-03-30T11:59:33.390 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-30T11:59:33.430 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-30T11:59:33.707 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-babel. 2024-03-30T11:59:33.734 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T11:59:33.758 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2024-03-30T11:59:33.773 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T11:59:33.785 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-30T11:59:33.824 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T11:59:34.051 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-jinja2. 2024-03-30T11:59:34.066 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-30T11:59:34.101 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T11:59:34.187 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-certifi. 2024-03-30T11:59:34.214 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-30T11:59:34.254 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-30T11:59:34.420 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-30T11:59:34.436 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:34.470 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:34.549 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-idna. 2024-03-30T11:59:34.576 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-30T11:59:34.607 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-30T11:59:34.766 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-30T11:59:34.793 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-30T11:59:34.832 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-30T11:59:34.893 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-urllib3. 2024-03-30T11:59:34.921 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-30T11:59:34.968 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T11:59:35.135 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-routes. 2024-03-30T11:59:35.163 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-30T11:59:35.202 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T11:59:35.288 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests. 2024-03-30T11:59:35.314 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-30T11:59:35.354 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T11:59:35.555 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-30T11:59:35.582 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:35.622 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:35.666 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-30T11:59:35.693 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-30T11:59:35.732 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T11:59:36.002 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-30T11:59:36.029 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:36.059 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-30T11:59:36.068 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:36.074 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-30T11:59:36.117 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T11:59:36.440 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-30T11:59:36.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:36.507 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:36.612 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-joblib. 2024-03-30T11:59:36.640 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-30T11:59:36.679 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T11:59:36.768 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-30T11:59:36.783 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-30T11:59:36.826 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-30T11:59:37.049 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-30T11:59:37.077 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-30T11:59:37.116 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-30T11:59:37.129 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2024-03-30T11:59:37.156 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-30T11:59:37.195 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-30T11:59:37.402 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-sklearn. 2024-03-30T11:59:37.430 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-30T11:59:37.468 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T11:59:37.540 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2024-03-30T11:59:37.567 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:37.607 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:37.960 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2024-03-30T11:59:37.987 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:38.026 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:38.161 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-30T11:59:38.189 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:38.313 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:38.589 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2024-03-30T11:59:38.615 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:38.655 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:39.217 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-cachetools. 2024-03-30T11:59:39.244 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-30T11:59:39.283 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-30T11:59:39.545 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-30T11:59:39.572 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-30T11:59:39.611 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-30T11:59:39.766 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2024-03-30T11:59:39.793 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:39.833 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:39.948 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-30T11:59:39.976 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-30T11:59:40.015 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T11:59:40.119 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2024-03-30T11:59:40.146 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:40.186 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:40.326 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-rsa. 2024-03-30T11:59:40.353 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-30T11:59:40.393 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-30T11:59:40.584 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2024-03-30T11:59:40.610 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:40.714 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-google-auth. 2024-03-30T11:59:40.739 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:40.741 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-30T11:59:40.780 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-30T11:59:41.101 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-30T11:59:41.125 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2024-03-30T11:59:41.128 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-30T11:59:41.152 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-cephadm_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:41.192 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:41.212 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T11:59:41.511 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-30T11:59:41.538 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-30T11:59:41.578 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-30T11:59:41.673 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-websocket. 2024-03-30T11:59:41.697 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-30T11:59:41.738 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-30T11:59:41.880 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-30T11:59:41.896 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T11:59:41.939 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T11:59:42.032 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-30T11:59:42.060 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-30T11:59:42.099 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T11:59:42.738 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-30T11:59:42.754 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:42.796 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:42.969 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-babel. 2024-03-30T11:59:42.997 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T11:59:43.027 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T11:59:43.191 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-30T11:59:43.219 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-30T11:59:43.266 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T11:59:43.297 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2024-03-30T11:59:43.324 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-30T11:59:43.355 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T11:59:43.611 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-30T11:59:43.638 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T11:59:43.658 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-30T11:59:43.707 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T11:59:43.707 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:43.717 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:43.938 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package jq. 2024-03-30T11:59:43.954 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T11:59:43.996 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-30T11:59:44.053 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-30T11:59:44.080 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-30T11:59:44.119 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-30T11:59:44.308 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package socat. 2024-03-30T11:59:44.335 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-30T11:59:44.372 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2024-03-30T11:59:44.383 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-30T11:59:44.399 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-30T11:59:44.430 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T11:59:44.708 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-30T11:59:44.738 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package xmlstarlet. 2024-03-30T11:59:44.738 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:44.764 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-30T11:59:44.766 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:44.811 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-30T11:59:45.191 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-test. 2024-03-30T11:59:45.195 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-30T11:59:45.218 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:45.223 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-30T11:59:45.257 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:45.261 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T11:59:45.799 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2024-03-30T11:59:45.825 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-30T11:59:45.865 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T11:59:46.335 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-30T11:59:46.362 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-30T11:59:46.402 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-30T11:59:46.730 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2024-03-30T11:59:46.756 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-30T11:59:46.796 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T11:59:47.422 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-30T11:59:47.449 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:47.512 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package ceph-volume. 2024-03-30T11:59:47.539 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:47.573 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:47.578 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:47.874 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-30T11:59:47.902 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:47.940 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:48.294 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-30T11:59:48.321 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-30T11:59:48.360 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T11:59:48.518 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2024-03-30T11:59:48.545 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-30T11:59:48.585 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-30T11:59:48.673 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-30T11:59:48.701 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-30T11:59:48.740 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-30T11:59:48.896 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-30T11:59:48.923 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-30T11:59:48.963 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-30T11:59:49.094 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package nvme-cli. 2024-03-30T11:59:49.122 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-30T11:59:49.160 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T11:59:49.274 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-30T11:59:49.302 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-30T11:59:49.341 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T11:59:49.575 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package pkg-config. 2024-03-30T11:59:49.602 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-30T11:59:49.641 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T11:59:49.661 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2024-03-30T11:59:49.688 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-30T11:59:49.727 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-30T11:59:49.894 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-30T11:59:49.921 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-30T11:59:49.960 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-30T11:59:49.989 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2024-03-30T11:59:50.005 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-30T11:59:50.047 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-30T11:59:50.367 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-30T11:59:50.394 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-30T11:59:50.414 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-30T11:59:50.434 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T11:59:50.441 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-30T11:59:50.480 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-30T11:59:50.770 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2024-03-30T11:59:50.792 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pastescript. 2024-03-30T11:59:50.797 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-30T11:59:50.819 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-30T11:59:50.837 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-30T11:59:50.858 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-30T11:59:51.149 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-30T11:59:51.176 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-30T11:59:51.178 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pluggy. 2024-03-30T11:59:51.194 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-30T11:59:51.215 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T11:59:51.228 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-30T11:59:51.490 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-psutil. 2024-03-30T11:59:51.517 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-30T11:59:51.556 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-30T11:59:51.851 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-py. 2024-03-30T11:59:51.867 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-30T11:59:51.909 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-30T11:59:51.912 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-30T11:59:51.928 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:51.970 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:52.221 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pygments. 2024-03-30T11:59:52.237 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-30T11:59:52.279 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T11:59:52.298 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-30T11:59:52.325 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-30T11:59:52.365 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T11:59:52.651 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-30T11:59:52.667 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T11:59:52.701 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T11:59:52.759 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-30T11:59:52.787 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-30T11:59:52.825 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-30T11:59:52.937 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2024-03-30T11:59:52.964 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T11:59:53.004 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-30T11:59:53.153 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-toml. 2024-03-30T11:59:53.181 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-30T11:59:53.219 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-30T11:59:53.256 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2024-03-30T11:59:53.284 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-30T11:59:53.323 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-30T11:59:53.497 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-pytest. 2024-03-30T11:59:53.525 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-30T11:59:53.555 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T11:59:53.634 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2024-03-30T11:59:53.662 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-30T11:59:53.701 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-30T11:59:53.909 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package python3-simplejson. 2024-03-30T11:59:53.937 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-30T11:59:53.976 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-30T11:59:54.013 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2024-03-30T11:59:54.030 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:54.063 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:54.253 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-30T11:59:54.270 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-30T11:59:54.312 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-30T11:59:55.066 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package radosgw. 2024-03-30T11:59:55.095 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../116-radosgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:55.132 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:55.927 INFO:teuthology.orchestra.run.smithi178.stdout:Selecting previously unselected package rbd-fuse. 2024-03-30T11:59:55.956 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:55.985 INFO:teuthology.orchestra.run.smithi178.stdout:Unpacking rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:56.290 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-30T11:59:56.493 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2024-03-30T11:59:56.521 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T11:59:56.559 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:56.701 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T11:59:56.855 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-30T11:59:56.883 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T11:59:56.922 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:56.942 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T11:59:57.192 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-30T11:59:57.241 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-30T11:59:57.269 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-30T11:59:57.307 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T11:59:57.557 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-30T11:59:57.612 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-30T11:59:57.628 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-30T11:59:57.670 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-30T11:59:57.976 INFO:teuthology.orchestra.run.smithi178.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-30T11:59:58.033 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2024-03-30T11:59:58.035 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T11:59:58.060 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-30T11:59:58.099 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T11:59:58.519 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user cephadm....done 2024-03-30T11:59:58.538 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package pkg-config. 2024-03-30T11:59:58.565 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-30T11:59:58.604 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T11:59:58.667 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T11:59:58.849 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-30T11:59:58.865 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-30T11:59:58.898 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-30T11:59:58.935 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-30T11:59:59.170 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-30T11:59:59.296 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-30T11:59:59.335 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-30T11:59:59.362 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-30T11:59:59.401 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-30T11:59:59.555 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-30T11:59:59.654 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2024-03-30T11:59:59.682 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-30T11:59:59.712 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-30T11:59:59.823 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T11:59:59.950 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-30T12:00:00.024 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2024-03-30T12:00:00.052 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-30T12:00:00.091 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:00:00.251 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:00:00.402 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2024-03-30T12:00:00.429 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-30T12:00:00.468 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-30T12:00:00.620 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-30T12:00:00.797 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2024-03-30T12:00:00.824 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-30T12:00:00.863 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-30T12:00:00.888 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-30T12:00:01.156 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:00:01.183 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pygments. 2024-03-30T12:00:01.210 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-30T12:00:01.249 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:00:01.399 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:00:01.651 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:00:01.720 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-30T12:00:01.748 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-30T12:00:01.787 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:00:01.902 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:00:02.106 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-toml. 2024-03-30T12:00:02.134 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-30T12:00:02.170 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:02.172 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-30T12:00:02.429 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:00:02.475 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2024-03-30T12:00:02.503 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-30T12:00:02.542 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:00:02.555 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:00:02.664 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:00:02.765 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:00:02.889 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2024-03-30T12:00:02.917 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-30T12:00:02.955 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:00:03.025 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:00:03.225 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-30T12:00:03.241 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-30T12:00:03.283 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:00:03.406 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-30T12:00:03.464 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-30T12:00:03.528 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:00:03.654 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-30T12:00:04.004 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2024-03-30T12:00:04.005 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:00:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../116-radosgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:00:04.070 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:04.115 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-30T12:00:04.383 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:00:04.643 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:00:04.864 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2024-03-30T12:00:04.881 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:00:04.923 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:04.927 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up zip (3.0-12build2) ... 2024-03-30T12:00:05.053 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:00:05.235 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-30T12:00:05.494 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T12:00:05.620 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T12:00:05.714 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:00:05.887 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-30T12:00:05.982 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:00:06.099 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:00:06.219 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-30T12:00:06.225 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:00:06.518 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-30T12:00:06.637 INFO:teuthology.orchestra.run.smithi172.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-30T12:00:06.739 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:06.878 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-30T12:00:07.129 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:00:07.298 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2024-03-30T12:00:07.454 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T12:00:07.522 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:00:07.799 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:00:07.898 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-30T12:00:08.142 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:00:08.149 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-30T12:00:08.275 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-30T12:00:08.385 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:00:08.502 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:08.543 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-30T12:00:08.802 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T12:00:08.804 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-idna (3.3-1) ... 2024-03-30T12:00:08.903 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-30T12:00:09.155 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:00:09.197 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:00:09.557 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-30T12:00:09.817 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-30T12:00:10.093 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:00:10.319 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:00:10.344 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:00:10.571 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:10.578 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:00:10.688 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-30T12:00:10.846 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:00:10.949 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:00:11.074 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:00:11.122 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:11.201 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:00:11.407 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:00:11.485 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:00:11.541 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:00:11.642 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:00:11.760 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:00:11.820 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-30T12:00:12.019 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:00:12.071 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:12.197 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:00:12.406 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-30T12:00:12.443 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-30T12:00:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:00:12.522 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:00:12.649 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-30T12:00:12.760 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-30T12:00:13.016 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:00:13.019 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-30T12:00:13.125 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-30T12:00:13.279 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:00:13.385 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:00:13.530 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-30T12:00:13.653 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:00:13.789 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:00:13.915 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-30T12:00:13.962 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up zip (3.0-12build2) ... 2024-03-30T12:00:14.088 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:00:14.200 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:00:14.326 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:00:14.618 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:00:14.773 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:00:14.919 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:00:15.041 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:00:15.142 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:00:15.187 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:00:15.260 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:00:15.313 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:00:15.439 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:00:15.528 INFO:teuthology.orchestra.run.smithi178.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-30T12:00:15.569 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-30T12:00:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:00:15.883 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:00:15.954 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-30T12:00:16.222 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:00:16.268 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:00:16.528 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-30T12:00:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:00:16.653 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-30T12:00:16.883 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:00:16.913 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:17.039 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:00:17.217 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:00:17.373 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:00:17.477 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:00:17.603 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:17.666 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:00:17.896 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-idna (3.3-1) ... 2024-03-30T12:00:17.951 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-30T12:00:18.076 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-30T12:00:18.213 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:00:18.352 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-30T12:00:18.629 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:00:18.982 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:00:19.344 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:00:19.375 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-30T12:00:19.587 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:19.652 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:00:19.722 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-30T12:00:19.990 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:00:19.995 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:00:20.116 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:00:20.138 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:20.242 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:00:20.264 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:20.390 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:00:20.535 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:00:20.702 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:20.777 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-30T12:00:20.828 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:20.954 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-30T12:00:21.028 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:21.155 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:00:21.230 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:21.339 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:21.465 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:21.481 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:00:21.678 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:00:21.679 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:00:21.732 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-30T12:00:21.975 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-30T12:00:22.192 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:22.218 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:00:22.301 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:22.419 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:22.460 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-30T12:00:22.553 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:00:22.728 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:00:22.854 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-30T12:00:23.139 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:00:23.265 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:00:23.549 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:00:23.790 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:23.834 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:00:23.924 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:24.059 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:24.076 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:00:24.184 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:24.185 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:00:24.310 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:24.312 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:00:24.401 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-30T12:00:24.630 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:00:24.757 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:00:24.828 INFO:teuthology.orchestra.run.smithi178.stdout:Adding group ceph....done 2024-03-30T12:00:25.133 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:00:25.228 INFO:teuthology.orchestra.run.smithi178.stdout:Adding system user ceph....done 2024-03-30T12:00:25.398 INFO:teuthology.orchestra.run.smithi178.stdout:Setting system user ceph properties....done 2024-03-30T12:00:25.409 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-30T12:00:25.411 INFO:teuthology.orchestra.run.smithi178.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-30T12:00:25.535 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-30T12:00:25.545 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-30T12:00:25.786 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:00:25.854 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-30T12:00:25.920 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:00:26.255 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:00:26.361 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:26.521 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:26.564 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:00:26.832 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-30T12:00:26.978 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:00:26.979 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:00:27.312 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-30T12:00:27.422 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:27.609 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-30T12:00:27.870 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:00:28.102 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-30T12:00:28.214 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:00:28.549 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:28.615 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-30T12:00:28.747 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:00:28.747 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:00:28.884 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:00:29.189 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:29.244 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:00:29.379 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:29.465 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:00:29.465 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:00:29.505 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:29.631 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:00:29.975 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.086 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.092 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.202 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-30T12:00:30.344 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:00:30.345 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:00:30.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.596 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.722 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.855 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:30.917 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:00:30.917 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:00:30.964 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.115 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.332 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:00:31.333 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:00:31.415 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.516 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.642 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.716 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:31.768 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:00:31.901 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:32.160 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:32.311 INFO:teuthology.orchestra.run.smithi178.stdout:Setting up ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:32.664 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-30T12:00:32.773 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-30T12:00:32.922 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:32.960 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-30T12:00:33.048 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:33.174 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:33.300 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:33.426 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2024-03-30T12:00:33.993 INFO:teuthology.orchestra.run.smithi178.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-30T12:00:34.329 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2024-03-30T12:00:34.498 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2024-03-30T12:00:34.512 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-30T12:00:34.654 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-30T12:00:34.662 INFO:teuthology.orchestra.run.smithi178.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-30T12:00:34.668 DEBUG:teuthology.parallel:result is None 2024-03-30T12:00:35.019 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-30T12:00:35.502 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:35.628 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:36.112 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:00:36.113 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:00:36.588 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:36.938 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-30T12:00:37.390 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:37.614 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:00:37.615 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:00:38.058 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:38.267 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:00:38.267 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:00:38.727 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:38.996 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:00:38.996 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:00:39.470 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:39.596 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:39.739 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:39.938 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:00:39.938 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:00:40.398 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:40.566 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:40.708 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:40.859 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:00:41.204 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-30T12:00:41.313 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-30T12:00:41.466 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-30T12:00:42.567 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-30T12:00:43.202 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-30T12:00:43.208 DEBUG:teuthology.parallel:result is None 2024-03-30T12:00:43.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:00:43.356 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2024-03-30T12:00:43.375 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:00:43.375 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:00:43.375 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864-g5dc2ea30-1jammy is installed. 2024-03-30T12:00:43.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:00:43.602 DEBUG:teuthology.orchestra.run.smithi178:> dpkg-query -W -f '${Version}' ceph 2024-03-30T12:00:43.624 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:00:43.625 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:00:43.625 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864-g5dc2ea30-1jammy is installed. 2024-03-30T12:00:43.627 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T12:00:43.627 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:43.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T12:00:43.642 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:43.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T12:00:43.678 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T12:00:43.678 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:43.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T12:00:43.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T12:00:43.754 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:43.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T12:00:43.766 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T12:00:43.820 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T12:00:43.821 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:43.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T12:00:43.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T12:00:43.886 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:43.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T12:00:43.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T12:00:43.953 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T12:00:43.953 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:43.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T12:00:43.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T12:00:44.021 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:44.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T12:00:44.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T12:00:44.089 INFO:teuthology.run_tasks:Running task ceph... 2024-03-30T12:00:44.188 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-30T12:00:44.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-03-30T12:00:44.191 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-03-30T12:00:44.203 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-30T12:00:44.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T12:00:44.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T12:00:44.261 INFO:tasks.ceph:Creating extra log directories... 2024-03-30T12:00:44.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T12:00:44.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T12:00:44.328 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-30T12:00:44.328 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-30T12:00:44.329 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612', 'branch': 'wip-ceph_test_rados-partial-reads', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7630612', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-ceph_test_rados-partial-reads'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ceph_test_rados-partial-reads', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9805, 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-ceph_test_rados-partial-reads', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'targets': {'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPFboCyO3bM9cLXOTIqjPA3FxZk5ahQ8MGtbCVb0ySQ79yoypJc3uYDzYtn5h4bDP1IlOWFsHQn5j7M2CKKkts=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPP32OwpGxEByXBvXQdTzC8LSFKogGaZO56cXXA/etDCnz6VFskXcEmmXaqaKJT5/21SSm5rA5Ikq1zBoq5ODBw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['ceph config set mgr mgr/balancer/log_level debug', 'ceph balancer status', 'ceph osd set-require-min-compat-client reef', 'ceph balancer mode upmap-read', 'ceph balancer on', 'ceph balancer status', 'ceph balancer status detail']}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 3600', 'sudo ceph osd pool set cache min_read_recency_for_promote 0', 'sudo ceph osd pool set cache min_write_recency_for_promote 0']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-30_00:41:04', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-30T12:00:44.329 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T12:00:44.356 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T12:00:44.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T12:00:44.404 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T12:00:44.427 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:44.427 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T12:00:44.463 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T12:00:44.463 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-30 11:55:22.624118488 +0000 2024-03-30T12:00:44.511 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-30 11:55:22.376120704 +0000 2024-03-30T12:00:44.512 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-30 11:55:22.376120704 +0000 2024-03-30T12:00:44.512 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-30T12:00:44.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T12:00:44.565 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-30T12:00:44.566 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-30T12:00:44.566 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000410416 s, 1.2 MB/s 2024-03-30T12:00:44.567 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T12:00:44.616 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-30T12:00:44.663 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T12:00:44.663 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:44.663 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-03-30T12:00:44.663 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:44.664 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-30 11:55:23.168113631 +0000 2024-03-30T12:00:44.664 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-30 11:55:22.888116130 +0000 2024-03-30T12:00:44.664 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-30 11:55:22.888116130 +0000 2024-03-30T12:00:44.664 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-30T12:00:44.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T12:00:44.717 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-30T12:00:44.717 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-30T12:00:44.717 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000417843 s, 1.2 MB/s 2024-03-30T12:00:44.719 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T12:00:44.768 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-30 11:55:23.668109164 +0000 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-30 11:55:23.420111380 +0000 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-30 11:55:23.420111380 +0000 2024-03-30T12:00:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-30T12:00:44.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T12:00:44.869 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-30T12:00:44.869 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-30T12:00:44.870 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000424193 s, 1.2 MB/s 2024-03-30T12:00:44.871 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T12:00:44.920 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-30 11:55:24.164104734 +0000 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-30 11:55:23.912106985 +0000 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-30 11:55:23.912106985 +0000 2024-03-30T12:00:44.967 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-30T12:00:44.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T12:00:45.021 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-30T12:00:45.021 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-30T12:00:45.021 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000312901 s, 1.6 MB/s 2024-03-30T12:00:45.022 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T12:00:45.072 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:45.072 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:45.072 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T12:00:45.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T12:00:45.078 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-03-30T12:00:45.127 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-30 11:55:21.566795735 +0000 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-30 11:55:21.306798057 +0000 2024-03-30T12:00:45.153 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-30 11:55:21.306798057 +0000 2024-03-30T12:00:45.154 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-03-30T12:00:45.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T12:00:45.180 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-30T12:00:45.180 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-30T12:00:45.180 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000327931 s, 1.6 MB/s 2024-03-30T12:00:45.182 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T12:00:45.231 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-03-30T12:00:45.277 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-30 11:55:22.062791305 +0000 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-30 11:55:21.790793734 +0000 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-30 11:55:21.790793734 +0000 2024-03-30T12:00:45.278 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-03-30T12:00:45.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T12:00:45.331 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-30T12:00:45.331 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-30T12:00:45.331 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000207513 s, 2.5 MB/s 2024-03-30T12:00:45.332 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T12:00:45.379 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-03-30T12:00:45.425 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-30 11:55:22.570786767 +0000 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-30 11:55:22.290789267 +0000 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-30 11:55:22.290789267 +0000 2024-03-30T12:00:45.426 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-03-30T12:00:45.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T12:00:45.480 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-30T12:00:45.481 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-30T12:00:45.481 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000205131 s, 2.5 MB/s 2024-03-30T12:00:45.482 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T12:00:45.531 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-30 11:55:23.070782301 +0000 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-30 11:55:22.818784552 +0000 2024-03-30T12:00:45.578 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-30 11:55:22.818784552 +0000 2024-03-30T12:00:45.579 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2024-03-30T12:00:45.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T12:00:45.631 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-30T12:00:45.631 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-30T12:00:45.631 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000199159 s, 2.6 MB/s 2024-03-30T12:00:45.633 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T12:00:45.683 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:45.683 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T12:00:45.683 INFO:tasks.ceph:Generating config... 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-30T12:00:45.685 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd beacon report interval = 30 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-03-30T12:00:45.686 INFO:tasks.ceph:[osd] osd map cache size = 1 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd max scrubs = 1 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-30T12:00:45.687 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2024-03-30T12:00:45.688 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-03-30T12:00:45.688 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-03-30T12:00:45.688 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-30T12:00:45.688 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2024-03-30T12:00:45.688 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] ms type = async 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-03-30T12:00:45.688 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-03-30T12:00:45.688 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-30T12:00:45.689 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-30T12:00:45.689 INFO:tasks.ceph:Setting up mon.a... 2024-03-30T12:00:45.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-30T12:00:45.724 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.keyring 2024-03-30T12:00:45.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-30T12:00:45.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:00:45.868 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.172'), ('mon.c', '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]'), ('mon.b', '172.21.15.178')] 2024-03-30T12:00:45.869 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790],172.21.15.178', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 1, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-30T12:00:45.870 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:45.870 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T12:00:45.915 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --addv c '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]' --add b 172.21.15.178 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 0958e16d-2906-41ba-a5c4-32c1d269fa2a 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = quincy 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 0958e16d-2906-41ba-a5c4-32c1d269fa2a 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2024-03-30T12:00:46.011535+0000 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:created 2024-03-30T12:00:46.011535+0000 2024-03-30T12:00:46.002 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 17 (quincy) 2024-03-30T12:00:46.003 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2024-03-30T12:00:46.003 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2024-03-30T12:00:46.003 INFO:teuthology.orchestra.run.smithi172.stdout:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-03-30T12:00:46.003 INFO:teuthology.orchestra.run.smithi172.stdout:2: [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon.c 2024-03-30T12:00:46.003 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-30T12:00:46.004 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T12:00:46.051 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0958e16d-2906-41ba-a5c4-32c1d269fa2a... 2024-03-30T12:00:46.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T12:00:46.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.125 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.126 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T12:00:46.127 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790],172.21.15.178" 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: mon election default strategy = 1 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 2500 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: ms tcp read timeout = 5 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: ms type = async 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-03-30T12:00:46.128 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_min_size = 2 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_size = 2 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 0958e16d-2906-41ba-a5c4-32c1d269fa2a 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 120 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-03-30T12:00:46.129 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-03-30T12:00:46.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = zstd 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore zero block detection = True 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-03-30T12:00:46.131 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd backoff on degraded = True 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd backoff on peering = True 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd beacon report interval = 30 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify cached snaps = True 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify missing on start = True 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd inject bad map crc probability = 0.1 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd map cache size = 1 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd max backfills = 6 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd max scrubs = 1 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock override recovery settings = True 2024-03-30T12:00:46.132 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue = debug_random 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue cut off = debug_random 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub during recovery = False 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub min interval = 60 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd_max_pg_log_entries = 3000 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: osd_min_pg_log_entries = 3000 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:00:46.133 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:00:46.134 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2024-03-30T12:00:46.135 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: mon scrub interval = 300 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2024-03-30T12:00:46.136 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-30T12:00:46.138 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.139 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-30T12:00:46.140 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.172,[v2:172.21.15.172:3301,v1:172.21.15.172:6790],172.21.15.178" 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon client directed command retry = 5 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: mon election default strategy = 1 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: ms inject socket failures = 2500 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: ms tcp read timeout = 5 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: ms type = async 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_min_size = 2 2024-03-30T12:00:46.141 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_size = 2 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 0958e16d-2906-41ba-a5c4-32c1d269fa2a 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 120 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-30T12:00:46.142 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.143 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug monc = 20 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore compression algorithm = zstd 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore compression mode = aggressive 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore zero block detection = True 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 20 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 20 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 10 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-30T12:00:46.144 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd backoff on degraded = True 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd backoff on peering = True 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd beacon report interval = 30 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd blocked scrub grace period = 3600 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify cached snaps = True 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify missing on start = True 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd inject bad map crc probability = 0.1 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd map cache size = 1 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd max backfills = 6 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd max markdown count = 1000 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd max scrubs = 1 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock override recovery settings = True 2024-03-30T12:00:46.145 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue = debug_random 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd op queue cut off = debug_random 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub during recovery = False 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub min interval = 60 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd shutdown pgref assert = True 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_pg_log_entries = 3000 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: osd_min_pg_log_entries = 3000 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-30T12:00:46.146 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-30T12:00:46.147 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon min osdmap epochs = 50 2024-03-30T12:00:46.148 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune interval = 2 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune min = 15 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: mon osdmap full prune txsize = 2 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: mon scrub interval = 300 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: paxos service trim min = 10 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-30T12:00:46.149 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-30T12:00:46.150 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-30T12:00:46.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-30T12:00:46.229 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-30T12:00:46.229 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:46.230 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-30T12:00:46.235 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:46.235 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-30T12:00:46.282 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:00:46.283 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:46.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T12:00:46.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:00:46.354 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:00:46.354 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:46.403 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:00:46.403 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:46.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T12:00:46.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:00:46.425 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:00:46.425 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:46.518 INFO:tasks.ceph:Setting up mon nodes... 2024-03-30T12:00:46.519 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-30T12:00:46.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T12:00:46.562 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T12:00:46.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T12:00:46.610 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T12:00:46.627 INFO:tasks.ceph:Setting up mds nodes... 2024-03-30T12:00:46.627 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-30T12:00:46.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T12:00:46.661 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-30T12:00:46.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T12:00:46.708 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-30T12:00:46.721 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-30T12:00:46.721 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T12:00:46.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-30T12:00:46.735 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:46.735 INFO:tasks.ceph:role: osd.0 2024-03-30T12:00:46.735 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:00:46.735 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T12:00:46.802 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:00:46.802 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:00:46.802 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:00:46.803 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:00:47.263 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-30T12:00:47.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-30T12:00:47.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-30T12:00:47.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-30T12:00:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:00:47.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:00:47.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-30T12:00:47.487 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:47.487 INFO:tasks.ceph:role: osd.1 2024-03-30T12:00:47.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:00:47.487 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T12:00:47.550 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:00:47.550 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:00:47.551 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:00:47.980 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-30T12:00:47.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-30T12:00:47.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-30T12:00:48.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-30T12:00:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:00:48.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:00:48.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-30T12:00:48.158 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:48.158 INFO:tasks.ceph:role: osd.2 2024-03-30T12:00:48.159 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:00:48.159 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T12:00:48.222 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:00:48.222 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:00:48.222 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:00:48.223 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:00:48.681 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-30T12:00:48.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-30T12:00:48.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-30T12:00:48.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-30T12:00:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:00:48.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:00:48.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-30T12:00:48.844 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:00:48.844 INFO:tasks.ceph:role: osd.3 2024-03-30T12:00:48.844 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:00:48.844 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T12:00:48.905 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:00:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:00:48.907 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:00:49.380 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-30T12:00:49.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-30T12:00:49.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-30T12:00:49.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-30T12:00:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:00:49.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:00:49.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:49.657 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.663+0000 7faa29b257c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-30T12:00:49.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.667+0000 7faa29b257c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-30T12:00:49.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.667+0000 7faa29b257c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:00:49.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.667+0000 7faa29b257c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:00:49.661 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.667+0000 7faa29b257c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:00:49.668 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:49.675+0000 7faa29b257c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-30T12:00:52.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T12:00:52.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:52.733 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.739+0000 7f81beab67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-30T12:00:52.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.739+0000 7f81beab67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-30T12:00:52.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.739+0000 7f81beab67c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:00:52.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.739+0000 7f81beab67c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:00:52.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.739+0000 7f81beab67c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:00:52.734 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:52.743+0000 7f81beab67c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-30T12:00:55.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T12:00:55.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.871+0000 7f87a30867c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.871+0000 7f87a30867c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.871+0000 7f87a30867c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.871+0000 7f87a30867c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.871+0000 7f87a30867c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:00:55.866 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:55.875+0000 7f87a30867c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-30T12:00:58.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T12:00:58.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:00:58.981 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-30T12:00:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-30T12:00:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:00:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:00:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:00:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-30T12:00:58.987+0000 7f472970c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-30T12:01:01.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T12:01:02.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-30T12:01:02.050 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:01:02.050 INFO:tasks.ceph:role: osd.4 2024-03-30T12:01:02.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:01:02.051 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T12:01:02.107 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:01:02.108 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:01:02.131 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-30T12:01:02.133 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-30T12:01:02.133 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-30T12:01:02.194 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-30T12:01:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:01:02.247 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:01:02.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-30T12:01:02.299 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:01:02.299 INFO:tasks.ceph:role: osd.5 2024-03-30T12:01:02.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:01:02.299 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T12:01:02.353 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:01:02.354 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:01:02.377 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-30T12:01:02.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-30T12:01:02.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-30T12:01:02.437 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-30T12:01:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:01:02.490 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:01:02.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-30T12:01:02.545 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:01:02.545 INFO:tasks.ceph:role: osd.6 2024-03-30T12:01:02.545 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:01:02.545 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T12:01:02.600 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:01:02.600 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:01:02.600 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:01:02.600 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:01:02.601 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:01:02.624 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-30T12:01:02.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-30T12:01:02.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-30T12:01:02.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-30T12:01:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:01:02.739 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:01:02.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-30T12:01:02.793 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:01:02.793 INFO:tasks.ceph:role: osd.7 2024-03-30T12:01:02.793 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:01:02.793 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T12:01:02.850 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:01:02.850 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:01:02.851 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:01:02.875 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-30T12:01:02.876 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-30T12:01:02.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-30T12:01:02.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-30T12:01:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:01:02.996 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:01:02.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:03.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:01:03.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:03.132+0000 7f3958f607c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-30T12:01:06.094 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T12:01:06.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-30T12:01:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-30T12:01:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:01:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:01:06.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:01:06.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:06.192+0000 7f4d38bd37c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-30T12:01:09.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T12:01:09.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:09.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-30T12:01:09.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-30T12:01:09.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:01:09.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:01:09.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:01:09.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:09.300+0000 7fb67ea837c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-30T12:01:12.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T12:01:12.339 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:12.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-30T12:01:12.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-30T12:01:12.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:01:12.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:01:12.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:01:12.420 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-30T12:01:12.424+0000 7f443812e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-30T12:01:15.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T12:01:15.464 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-30T12:01:15.465 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-30T12:01:15.479 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-30T12:01:15.534 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-30T12:01:15.589 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-30T12:01:15.641 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-30T12:01:15.693 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-30T12:01:15.707 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-30T12:01:15.761 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-30T12:01:15.816 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-30T12:01:15.883 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-30T12:01:15.937 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:15.937 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-30T12:01:15.943 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:15.943 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-30T12:01:15.985 INFO:tasks.ceph:Adding keys to all mons... 2024-03-30T12:01:15.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T12:01:15.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDu/gdmarEfIhAATSmOJGyasHbB4L06QyQg5g== 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDx/gdmpGPHJxAAecroaP9O8V6OExkg5uGu8A== 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD0/gdm99RILBAAcI1JzqBRXgFQJ4uE0GRhtQ== 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD3/gdmMY0sNBAABAxEN7sGrezEBQeOoMBNbw== 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD6/gdm5lQNOxAA2FynkroB1siRQACxMGFvmA== 2024-03-30T12:01:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDu/gdm9o3sJBAA0huPuWxlGI5aqDlAssuQ/g== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD//gdmDuQRCBAAWRHbfSW6ILjF2otVo0iSIA== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAC/wdmTjazCxAAYH9MW648RChWtFbVoR4yRA== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAF/wdmaW4kEhAA4ZJ091uIFyi+WiBtN0f/QA== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAI/wdm/6GIGRAAKdaS+ag4kyDLrU7eRQR9ew== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDu/gdm+QgGKBAAoHNPY0h5V+3A0OyW0zJOGw== 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-03-30T12:01:16.033 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDu/gdmfljGKhAAswROffjoJdZ5pP0JSSFF5Q== 2024-03-30T12:01:16.039 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.y] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDu/gdmarEfIhAATSmOJGyasHbB4L06QyQg5g== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDx/gdmpGPHJxAAecroaP9O8V6OExkg5uGu8A== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD0/gdm99RILBAAcI1JzqBRXgFQJ4uE0GRhtQ== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD3/gdmMY0sNBAABAxEN7sGrezEBQeOoMBNbw== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.3] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD6/gdm5lQNOxAA2FynkroB1siRQACxMGFvmA== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDu/gdm9o3sJBAA0huPuWxlGI5aqDlAssuQ/g== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.4] 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQD//gdmDuQRCBAAWRHbfSW6ILjF2otVo0iSIA== 2024-03-30T12:01:16.040 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.5] 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAC/wdmTjazCxAAYH9MW648RChWtFbVoR4yRA== 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.6] 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAF/wdmaW4kEhAA4ZJ091uIFyi+WiBtN0f/QA== 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.7] 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAI/wdm/6GIGRAAKdaS+ag4kyDLrU7eRQR9ew== 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDu/gdm+QgGKBAAoHNPY0h5V+3A0OyW0zJOGw== 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2024-03-30T12:01:16.041 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQDu/gdmfljGKhAAswROffjoJdZ5pP0JSSFF5Q== 2024-03-30T12:01:16.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:01:16.079 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:01:16.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.222 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:01:16.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:01:16.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.635 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:01:16.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:01:16.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:01:16.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:01:16.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:01:17.022 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-30T12:01:17.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-30T12:01:17.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:01:17.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-30T12:01:17.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-30T12:01:17.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:01:17.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-30T12:01:17.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-30T12:01:17.364 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:01:17.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-30T12:01:17.465 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:17.467 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:01:17.513 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-30T12:01:17.513 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-30T12:01:17.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-30T12:01:17.515 INFO:tasks.ceph.mon.a:Started 2024-03-30T12:01:17.515 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-30T12:01:17.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-30T12:01:17.518 INFO:tasks.ceph.mon.c:Started 2024-03-30T12:01:17.518 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-30T12:01:17.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-30T12:01:17.558 INFO:tasks.ceph.mon.b:Started 2024-03-30T12:01:17.559 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-30T12:01:17.559 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-30T12:01:17.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-30T12:01:17.562 INFO:tasks.ceph.mgr.y:Started 2024-03-30T12:01:17.562 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-30T12:01:17.562 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-30T12:01:17.564 INFO:tasks.ceph.mgr.x:Started 2024-03-30T12:01:17.564 INFO:tasks.ceph:Setting crush tunables to default 2024-03-30T12:01:17.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd crush tunables default 2024-03-30T12:01:23.143 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:23.151+0000 7efc38453180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T12:01:23.211 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:23.216+0000 7f10bc089180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T12:01:23.662 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:23.671+0000 7efc38453180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T12:01:23.728 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:23.732+0000 7f10bc089180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T12:01:23.780 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2024-03-30T12:01:23.796 INFO:tasks.ceph:check_enable_crimson: False 2024-03-30T12:01:23.796 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-30T12:01:23.797 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:23.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-30T12:01:23.808 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:23.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-30T12:01:23.864 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:23.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-30T12:01:23.873 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:23.879+0000 7efc38453180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T12:01:23.883 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-30T12:01:23.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-30T12:01:23.919 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:23.924+0000 7f10bc089180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T12:01:23.938 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:23.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-30T12:01:23.950 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:23.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-30T12:01:24.004 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:24.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-30T12:01:24.059 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-30T12:01:24.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-30T12:01:24.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 076ecae5-da19-444a-bd06-667256ac89ed 0 2024-03-30T12:01:24.369 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-03-30T12:01:24.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 0ad37a2f-4ebf-4265-8003-f35267f1d7f3 1 2024-03-30T12:01:24.425 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.431+0000 7efc38453180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T12:01:24.478 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.484+0000 7f10bc089180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T12:01:24.479 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.487+0000 7efc38453180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T12:01:24.532 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.539+0000 7efc38453180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T12:01:24.536 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.540+0000 7f10bc089180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T12:01:24.577 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-03-30T12:01:24.590 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.596+0000 7f10bc089180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T12:01:24.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new fad1e8ec-c235-45cc-8d4a-40d567b40e74 2 2024-03-30T12:01:24.610 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.619+0000 7efc38453180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T12:01:24.662 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.671+0000 7efc38453180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T12:01:24.670 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.676+0000 7f10bc089180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T12:01:24.724 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.728+0000 7f10bc089180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T12:01:24.739 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.747+0000 7efc38453180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T12:01:24.778 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-03-30T12:01:24.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 24a3b726-2aee-4271-a035-592516efcec2 3 2024-03-30T12:01:24.795 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.803+0000 7efc38453180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T12:01:24.803 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.808+0000 7f10bc089180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T12:01:24.858 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:24.864+0000 7f10bc089180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T12:01:24.944 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:24.951+0000 7efc38453180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T12:01:24.995 INFO:teuthology.orchestra.run.smithi178.stdout:3 2024-03-30T12:01:25.007 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:25.015+0000 7efc38453180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T12:01:25.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 9b998b21-6f34-444b-9a0d-c8923b3af8d1 4 2024-03-30T12:01:25.012 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:25.016+0000 7f10bc089180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T12:01:25.075 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:25.080+0000 7f10bc089180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T12:01:25.200 INFO:teuthology.orchestra.run.smithi178.stdout:4 2024-03-30T12:01:25.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 140ae929-bb76-4f4d-a85c-d4c5e8f45562 5 2024-03-30T12:01:25.457 INFO:teuthology.orchestra.run.smithi178.stdout:5 2024-03-30T12:01:25.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6 6 2024-03-30T12:01:25.661 INFO:teuthology.orchestra.run.smithi178.stdout:6 2024-03-30T12:01:25.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 0a436e85-027e-417c-bb92-3871492fb39c 7 2024-03-30T12:01:25.685 INFO:tasks.ceph.mgr.x.smithi178.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-30T12:01:25.685 INFO:tasks.ceph.mgr.x.smithi178.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-30T12:01:25.685 INFO:tasks.ceph.mgr.x.smithi178.stderr: from numpy import show_config as show_numpy_config 2024-03-30T12:01:25.693 INFO:tasks.ceph.mgr.y.smithi172.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-30T12:01:25.693 INFO:tasks.ceph.mgr.y.smithi172.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-30T12:01:25.693 INFO:tasks.ceph.mgr.y.smithi172.stderr: from numpy import show_config as show_numpy_config 2024-03-30T12:01:25.731 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:25.736+0000 7f10bc089180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T12:01:25.783 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:25.791+0000 7efc38453180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T12:01:25.866 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:25.872+0000 7f10bc089180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:01:25.873 INFO:teuthology.orchestra.run.smithi178.stdout:7 2024-03-30T12:01:25.889 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-30T12:01:25.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-30T12:01:25.892 INFO:tasks.ceph.osd.0:Started 2024-03-30T12:01:25.892 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T12:01:25.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T12:01:25.895 INFO:tasks.ceph.osd.1:Started 2024-03-30T12:01:25.895 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-30T12:01:25.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-30T12:01:25.898 INFO:tasks.ceph.osd.2:Started 2024-03-30T12:01:25.898 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T12:01:25.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-30T12:01:25.900 INFO:tasks.ceph.osd.3:Started 2024-03-30T12:01:25.900 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T12:01:25.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T12:01:25.934 INFO:tasks.ceph.osd.4:Started 2024-03-30T12:01:25.934 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T12:01:25.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T12:01:25.937 INFO:tasks.ceph.osd.5:Started 2024-03-30T12:01:25.937 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T12:01:25.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T12:01:25.939 INFO:tasks.ceph.osd.6:Started 2024-03-30T12:01:25.939 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T12:01:25.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T12:01:25.941 INFO:tasks.ceph.osd.7:Started 2024-03-30T12:01:25.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:25.999 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:26.007+0000 7efc38453180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:01:26.055 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:26.060+0000 7f10bc089180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T12:01:26.123 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:26.128+0000 7f10bc089180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T12:01:26.131 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:26.131 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":10,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:25.879162+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:26.147 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-30T12:01:26.147 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-30T12:01:26.161 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:26.167+0000 7efc38453180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T12:01:26.225 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:26.231+0000 7efc38453180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T12:01:26.268 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:26.272+0000 7f10bc089180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:01:26.359 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:26.367+0000 7efc38453180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:01:26.522 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-30T12:01:26.531+0000 7f47db87e7c0 -1 Falling back to public interface 2024-03-30T12:01:26.527 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:26.535+0000 7f1a5d2ec7c0 -1 Falling back to public interface 2024-03-30T12:01:26.530 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:26.539+0000 7f68bf6807c0 -1 Falling back to public interface 2024-03-30T12:01:26.530 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:26.539+0000 7f5e185217c0 -1 Falling back to public interface 2024-03-30T12:01:26.531 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:26.539+0000 7f68be640640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6822/0: (98) Address already in use 2024-03-30T12:01:26.563 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-30T12:01:26.568+0000 7efdd75e17c0 -1 Falling back to public interface 2024-03-30T12:01:26.577 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-30T12:01:26.584+0000 7f189398a7c0 -1 Falling back to public interface 2024-03-30T12:01:26.579 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:26.584+0000 7fe2d21c57c0 -1 Falling back to public interface 2024-03-30T12:01:26.585 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:26.592+0000 7f8eaf1957c0 -1 Falling back to public interface 2024-03-30T12:01:26.887 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:26.892+0000 7f10bc089180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T12:01:26.953 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:26.963 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:26.968+0000 7f10bc089180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T12:01:26.978 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:26.987+0000 7efc38453180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T12:01:27.056 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:27.063+0000 7efc38453180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T12:01:27.135 INFO:teuthology.misc.health.smithi172.stdout: 2024-03-30T12:01:27.135 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":10,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:25.879162+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:27.141 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:27.148+0000 7f10bc089180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T12:01:27.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T12:01:27.214 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:27.220+0000 7f10bc089180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T12:01:27.243 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:27.251+0000 7efc38453180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T12:01:27.317 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:27.323+0000 7efc38453180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T12:01:27.497 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:27.504+0000 7f10bc089180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T12:01:27.562 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:27.564+0000 7f10bc089180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T12:01:27.609 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:27.615+0000 7efc38453180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T12:01:27.674 INFO:tasks.ceph.mgr.y.smithi172.stderr:2024-03-30T12:01:27.683+0000 7efc38453180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T12:01:28.600 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:28.604+0000 7f10634d1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-30T12:01:28.601 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-30T12:01:28.608+0000 7f10634d1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-30T12:01:30.326 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-30T12:01:30.332+0000 7efdd75e17c0 -1 osd.6 0 log_to_monitors true 2024-03-30T12:01:30.348 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-30T12:01:30.355+0000 7f47db87e7c0 -1 osd.0 0 log_to_monitors true 2024-03-30T12:01:30.377 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:30.383+0000 7f5e185217c0 -1 osd.2 0 log_to_monitors true 2024-03-30T12:01:30.420 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:30.424+0000 7f8eaf1957c0 -1 osd.7 0 log_to_monitors true 2024-03-30T12:01:30.421 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:30.427+0000 7f68bf6807c0 -1 osd.3 0 log_to_monitors true 2024-03-30T12:01:30.427 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:30.432+0000 7fe2d21c57c0 -1 osd.5 0 log_to_monitors true 2024-03-30T12:01:30.428 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:30.435+0000 7f1a5d2ec7c0 -1 osd.1 0 log_to_monitors true 2024-03-30T12:01:30.488 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-30T12:01:30.492+0000 7f189398a7c0 -1 osd.4 0 log_to_monitors true 2024-03-30T12:01:31.608 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-30T12:01:31.612+0000 7f188f933640 -1 osd.4 0 waiting for initial osdmap 2024-03-30T12:01:31.608 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-30T12:01:31.612+0000 7f1887310640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.610 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:31.616+0000 7f8eab13e640 -1 osd.7 0 waiting for initial osdmap 2024-03-30T12:01:31.610 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:31.619+0000 7f5e144ca640 -1 osd.2 0 waiting for initial osdmap 2024-03-30T12:01:31.611 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:31.616+0000 7fe2ce16e640 -1 osd.5 0 waiting for initial osdmap 2024-03-30T12:01:31.611 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:31.619+0000 7f1a59295640 -1 osd.1 0 waiting for initial osdmap 2024-03-30T12:01:31.612 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:31.619+0000 7f68bb629640 -1 osd.3 0 waiting for initial osdmap 2024-03-30T12:01:31.615 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:31.620+0000 7f8ea2b1b640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.615 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:31.620+0000 7fe2c5b4b640 -1 osd.5 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.617 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:31.623+0000 7f5e0bea7640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.618 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:31.627+0000 7f1a50c72640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.619 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:31.627+0000 7f68b3006640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:31.628 INFO:tasks.ceph.osd.3.smithi172.stderr:2024-03-30T12:01:31.635+0000 7f68b7466640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:31.638 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:31.647+0000 7f1a550d2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:31.639 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:31.647+0000 7f5e10307640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:31.642 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-30T12:01:31.648+0000 7f188b770640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:31.643 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:31.648+0000 7fe2c9fab640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:31.644 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:31.648+0000 7f8ea6f7b640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:32.274 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-30T12:01:32.280+0000 7efdd358a640 -1 osd.6 0 waiting for initial osdmap 2024-03-30T12:01:32.275 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-30T12:01:32.280+0000 7efdc6f5f640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:32.296 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-30T12:01:32.300+0000 7efdcb3bf640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:32.323 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-30T12:01:32.331+0000 7f47d7827640 -1 osd.0 0 waiting for initial osdmap 2024-03-30T12:01:32.324 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-30T12:01:32.331+0000 7f47cb1fc640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-03-30T12:01:32.343 INFO:tasks.ceph.osd.0.smithi172.stderr:2024-03-30T12:01:32.351+0000 7f47cf65c640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:32.635 INFO:tasks.ceph.osd.2.smithi172.stderr:2024-03-30T12:01:32.643+0000 7f5e0bea7640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:01:33.958 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:34.257 INFO:teuthology.misc.health.smithi172.stdout: 2024-03-30T12:01:34.257 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":14,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:33.631561+0000","last_up_change":"2024-03-30T12:01:32.619553+0000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6801","nonce":2789107731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6803","nonce":2789107731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6807","nonce":2789107731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6805","nonce":2789107731}]},"public_addr":"172.21.15.172:6801/2789107731","cluster_addr":"172.21.15.172:6803/2789107731","heartbeat_back_addr":"172.21.15.172:6807/2789107731","heartbeat_front_addr":"172.21.15.172:6805/2789107731","state":["exists","up"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6809","nonce":307083902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6811","nonce":307083902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6815","nonce":307083902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6813","nonce":307083902}]},"public_addr":"172.21.15.172:6809/307083902","cluster_addr":"172.21.15.172:6811/307083902","heartbeat_back_addr":"172.21.15.172:6815/307083902","heartbeat_front_addr":"172.21.15.172:6813/307083902","state":["exists","up"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6819","nonce":3212069946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6821","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6822","nonce":3212069946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6831","nonce":3212069946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6826","nonce":3212069946}]},"public_addr":"172.21.15.172:6819/3212069946","cluster_addr":"172.21.15.172:6822/3212069946","heartbeat_back_addr":"172.21.15.172:6831/3212069946","heartbeat_front_addr":"172.21.15.172:6826/3212069946","state":["exists","up"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6818","nonce":1025508895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6823","nonce":1025508895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6829","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6830","nonce":1025508895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6825","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6827","nonce":1025508895}]},"public_addr":"172.21.15.172:6818/1025508895","cluster_addr":"172.21.15.172:6823/1025508895","heartbeat_back_addr":"172.21.15.172:6830/1025508895","heartbeat_front_addr":"172.21.15.172:6827/1025508895","state":["exists","up"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6809","nonce":3222045438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6811","nonce":3222045438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6817","nonce":3222045438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6813","nonce":3222045438}]},"public_addr":"172.21.15.178:6809/3222045438","cluster_addr":"172.21.15.178:6811/3222045438","heartbeat_back_addr":"172.21.15.178:6817/3222045438","heartbeat_front_addr":"172.21.15.178:6813/3222045438","state":["exists","up"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6816","nonce":3461836880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6819","nonce":3461836880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6823","nonce":3461836880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6821","nonce":3461836880}]},"public_addr":"172.21.15.178:6816/3461836880","cluster_addr":"172.21.15.178:6819/3461836880","heartbeat_back_addr":"172.21.15.178:6823/3461836880","heartbeat_front_addr":"172.21.15.178:6821/3461836880","state":["exists","up"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6801","nonce":495238561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6803","nonce":495238561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6807","nonce":495238561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6805","nonce":495238561}]},"public_addr":"172.21.15.178:6801/495238561","cluster_addr":"172.21.15.178:6803/495238561","heartbeat_back_addr":"172.21.15.178:6807/495238561","heartbeat_front_addr":"172.21.15.178:6805/495238561","state":["exists","up"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6825","nonce":2824579992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6827","nonce":2824579992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6831","nonce":2824579992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6829","nonce":2824579992}]},"public_addr":"172.21.15.178:6825/2824579992","cluster_addr":"172.21.15.178:6827/2824579992","heartbeat_back_addr":"172.21.15.178:6831/2824579992","heartbeat_front_addr":"172.21.15.178:6829/2824579992","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.353563+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.465604+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.409253+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.429436+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.518286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.469883+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.366013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.437737+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:34.275 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-30T12:01:34.275 INFO:tasks.ceph:Creating RBD pool 2024-03-30T12:01:34.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-30T12:01:34.636 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2024-03-30T12:01:34.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-30T12:01:35.283 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:35.288+0000 7fe2c3b47640 -1 osd.5 13 handle_osd_map injecting map crc failure 2024-03-30T12:01:35.398 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-30T12:01:35.404+0000 7f8ea2b1b640 -1 osd.7 13 handle_osd_map injecting map crc failure 2024-03-30T12:01:35.538 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:35.547+0000 7f1a50c72640 -1 osd.1 13 handle_osd_map injecting map crc failure 2024-03-30T12:01:35.640 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-30T12:01:35.649 INFO:tasks.ceph.osd.1.smithi172.stderr:2024-03-30T12:01:35.655+0000 7f1a50c72640 -1 osd.1 15 handle_osd_map injecting map crc failure 2024-03-30T12:01:35.668 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-30T12:01:35.669 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T12:01:35.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-30T12:01:35.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-30T12:01:35.669 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-30T12:01:36.026 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:36.046 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1590853466},{"type":"v1","addr":"172.21.15.178:6833","nonce":1590853466}]},"active_addr":"172.21.15.178:6833/1590853466","active_change":"2024-03-30T12:01:27.590242+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":832764989}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1474648209}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":4074607801}]}]} 2024-03-30T12:01:36.050 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-30T12:01:36.050 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-30T12:01:36.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:36.350 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":16,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:35.642885+0000","last_up_change":"2024-03-30T12:01:32.619553+0000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T12:01:33.686010+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T12:01:34.619044+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6801","nonce":2789107731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6803","nonce":2789107731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6807","nonce":2789107731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6805","nonce":2789107731}]},"public_addr":"172.21.15.172:6801/2789107731","cluster_addr":"172.21.15.172:6803/2789107731","heartbeat_back_addr":"172.21.15.172:6807/2789107731","heartbeat_front_addr":"172.21.15.172:6805/2789107731","state":["exists","up"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6809","nonce":307083902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6811","nonce":307083902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6815","nonce":307083902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6813","nonce":307083902}]},"public_addr":"172.21.15.172:6809/307083902","cluster_addr":"172.21.15.172:6811/307083902","heartbeat_back_addr":"172.21.15.172:6815/307083902","heartbeat_front_addr":"172.21.15.172:6813/307083902","state":["exists","up"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6819","nonce":3212069946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6821","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6822","nonce":3212069946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6831","nonce":3212069946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6826","nonce":3212069946}]},"public_addr":"172.21.15.172:6819/3212069946","cluster_addr":"172.21.15.172:6822/3212069946","heartbeat_back_addr":"172.21.15.172:6831/3212069946","heartbeat_front_addr":"172.21.15.172:6826/3212069946","state":["exists","up"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6818","nonce":1025508895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6823","nonce":1025508895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6829","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6830","nonce":1025508895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6825","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6827","nonce":1025508895}]},"public_addr":"172.21.15.172:6818/1025508895","cluster_addr":"172.21.15.172:6823/1025508895","heartbeat_back_addr":"172.21.15.172:6830/1025508895","heartbeat_front_addr":"172.21.15.172:6827/1025508895","state":["exists","up"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6809","nonce":3222045438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6811","nonce":3222045438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6817","nonce":3222045438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6813","nonce":3222045438}]},"public_addr":"172.21.15.178:6809/3222045438","cluster_addr":"172.21.15.178:6811/3222045438","heartbeat_back_addr":"172.21.15.178:6817/3222045438","heartbeat_front_addr":"172.21.15.178:6813/3222045438","state":["exists","up"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6816","nonce":3461836880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6819","nonce":3461836880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6823","nonce":3461836880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6821","nonce":3461836880}]},"public_addr":"172.21.15.178:6816/3461836880","cluster_addr":"172.21.15.178:6819/3461836880","heartbeat_back_addr":"172.21.15.178:6823/3461836880","heartbeat_front_addr":"172.21.15.178:6821/3461836880","state":["exists","up"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6801","nonce":495238561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6803","nonce":495238561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6807","nonce":495238561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6805","nonce":495238561}]},"public_addr":"172.21.15.178:6801/495238561","cluster_addr":"172.21.15.178:6803/495238561","heartbeat_back_addr":"172.21.15.178:6807/495238561","heartbeat_front_addr":"172.21.15.178:6805/495238561","state":["exists","up"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6825","nonce":2824579992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6827","nonce":2824579992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6831","nonce":2824579992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6829","nonce":2824579992}]},"public_addr":"172.21.15.178:6825/2824579992","cluster_addr":"172.21.15.178:6827/2824579992","heartbeat_back_addr":"172.21.15.178:6831/2824579992","heartbeat_front_addr":"172.21.15.178:6829/2824579992","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.353563+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.465604+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.409253+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.429436+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.518286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.469883+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.366013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.437737+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:36.371 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-30T12:01:36.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:36.662 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:36.662 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":17,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:36.649840+0000","last_up_change":"2024-03-30T12:01:32.619553+0000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T12:01:33.686010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T12:01:34.619044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6801","nonce":2789107731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6803","nonce":2789107731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6807","nonce":2789107731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6805","nonce":2789107731}]},"public_addr":"172.21.15.172:6801/2789107731","cluster_addr":"172.21.15.172:6803/2789107731","heartbeat_back_addr":"172.21.15.172:6807/2789107731","heartbeat_front_addr":"172.21.15.172:6805/2789107731","state":["exists","up"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6809","nonce":307083902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6811","nonce":307083902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6815","nonce":307083902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6813","nonce":307083902}]},"public_addr":"172.21.15.172:6809/307083902","cluster_addr":"172.21.15.172:6811/307083902","heartbeat_back_addr":"172.21.15.172:6815/307083902","heartbeat_front_addr":"172.21.15.172:6813/307083902","state":["exists","up"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6819","nonce":3212069946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6821","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6822","nonce":3212069946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6831","nonce":3212069946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6826","nonce":3212069946}]},"public_addr":"172.21.15.172:6819/3212069946","cluster_addr":"172.21.15.172:6822/3212069946","heartbeat_back_addr":"172.21.15.172:6831/3212069946","heartbeat_front_addr":"172.21.15.172:6826/3212069946","state":["exists","up"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6818","nonce":1025508895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6823","nonce":1025508895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6829","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6830","nonce":1025508895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6825","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6827","nonce":1025508895}]},"public_addr":"172.21.15.172:6818/1025508895","cluster_addr":"172.21.15.172:6823/1025508895","heartbeat_back_addr":"172.21.15.172:6830/1025508895","heartbeat_front_addr":"172.21.15.172:6827/1025508895","state":["exists","up"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6809","nonce":3222045438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6811","nonce":3222045438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6817","nonce":3222045438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6813","nonce":3222045438}]},"public_addr":"172.21.15.178:6809/3222045438","cluster_addr":"172.21.15.178:6811/3222045438","heartbeat_back_addr":"172.21.15.178:6817/3222045438","heartbeat_front_addr":"172.21.15.178:6813/3222045438","state":["exists","up"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6816","nonce":3461836880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6819","nonce":3461836880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6823","nonce":3461836880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6821","nonce":3461836880}]},"public_addr":"172.21.15.178:6816/3461836880","cluster_addr":"172.21.15.178:6819/3461836880","heartbeat_back_addr":"172.21.15.178:6823/3461836880","heartbeat_front_addr":"172.21.15.178:6821/3461836880","state":["exists","up"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6801","nonce":495238561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6803","nonce":495238561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6807","nonce":495238561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6805","nonce":495238561}]},"public_addr":"172.21.15.178:6801/495238561","cluster_addr":"172.21.15.178:6803/495238561","heartbeat_back_addr":"172.21.15.178:6807/495238561","heartbeat_front_addr":"172.21.15.178:6805/495238561","state":["exists","up"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6825","nonce":2824579992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6827","nonce":2824579992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6831","nonce":2824579992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6829","nonce":2824579992}]},"public_addr":"172.21.15.178:6825/2824579992","cluster_addr":"172.21.15.178:6827/2824579992","heartbeat_back_addr":"172.21.15.178:6831/2824579992","heartbeat_front_addr":"172.21.15.178:6829/2824579992","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.353563+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.465604+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.409253+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.429436+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.518286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.469883+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.366013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.437737+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:36.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:36.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:36.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-30T12:01:36.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-30T12:01:36.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-30T12:01:36.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-30T12:01:36.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-30T12:01:36.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-30T12:01:36.951 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:36.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T12:01:36.957 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:36.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T12:01:36.958 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:36.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T12:01:36.967 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:36.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T12:01:36.974 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:36.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T12:01:36.975 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:36.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T12:01:36.977 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:36.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T12:01:37.066 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:37.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T12:01:37.374 INFO:teuthology.orchestra.run.smithi172.stdout:55834574849 2024-03-30T12:01:37.406 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.4 2024-03-30T12:01:37.459 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:37.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2024-03-30T12:01:37.491 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:37.491 INFO:teuthology.orchestra.run.smithi172.stdout:55834574849 2024-03-30T12:01:37.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.7 2024-03-30T12:01:37.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2024-03-30T12:01:37.550 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:37.551 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:37.563 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:37.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.5 2024-03-30T12:01:37.578 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-03-30T12:01:37.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2024-03-30T12:01:37.591 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:37.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.1 2024-03-30T12:01:38.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T12:01:38.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T12:01:38.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T12:01:38.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T12:01:38.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T12:01:38.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T12:01:38.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T12:01:38.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T12:01:38.793 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:38.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.4 2024-03-30T12:01:38.820 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:38.892 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:38.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2024-03-30T12:01:38.911 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.036 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:39.053 INFO:teuthology.orchestra.run.smithi172.stdout:55834574850 2024-03-30T12:01:39.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2024-03-30T12:01:39.059 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.7 2024-03-30T12:01:39.071 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.088 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:39.096 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:39.102 INFO:teuthology.orchestra.run.smithi172.stdout:55834574851 2024-03-30T12:01:39.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2024-03-30T12:01:39.113 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.118 INFO:teuthology.orchestra.run.smithi172.stdout:55834574852 2024-03-30T12:01:39.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2024-03-30T12:01:39.122 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2024-03-30T12:01:39.123 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.135 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-03-30T12:01:39.135 DEBUG:teuthology.parallel:result is None 2024-03-30T12:01:39.136 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T12:01:39.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:01:39.440 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:39.440 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:01:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-30T12:01:37.601904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221440,"kb_used_data":1068,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753280,"statfs":{"total":773094113280,"available":772867358720,"internally_reserved":0,"allocated":1093632,"data_stored":1145306,"data_compressed":6128,"data_compressed_allocated":49152,"data_compressed_original":778240,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793282+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:36.793282+0000","last_peered":"2024-03-30T12:01:36.793282+0000","last_clean":"2024-03-30T12:01:36.793282+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:36.793282+0000","last_undegraded":"2024-03-30T12:01:36.793282+0000","last_fullsized":"2024-03-30T12:01:36.793282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.656496+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:36.656496+0000","last_peered":"2024-03-30T12:01:36.656496+0000","last_clean":"2024-03-30T12:01:36.656496+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:36.656496+0000","last_undegraded":"2024-03-30T12:01:36.656496+0000","last_fullsized":"2024-03-30T12:01:36.656496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793315+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:36.793315+0000","last_peered":"2024-03-30T12:01:36.793315+0000","last_clean":"2024-03-30T12:01:36.793315+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:36.793315+0000","last_undegraded":"2024-03-30T12:01:36.793315+0000","last_fullsized":"2024-03-30T12:01:36.793315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793201+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:36.793201+0000","last_peered":"2024-03-30T12:01:36.793201+0000","last_clean":"2024-03-30T12:01:36.793201+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:36.793201+0000","last_undegraded":"2024-03-30T12:01:36.793201+0000","last_fullsized":"2024-03-30T12:01:36.793201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793242+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:36.793242+0000","last_peered":"2024-03-30T12:01:36.793242+0000","last_clean":"2024-03-30T12:01:36.793242+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:36.793242+0000","last_undegraded":"2024-03-30T12:01:36.793242+0000","last_fullsized":"2024-03-30T12:01:36.793242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.791741+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:36.791741+0000","last_peered":"2024-03-30T12:01:36.791741+0000","last_clean":"2024-03-30T12:01:36.791741+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:36.791741+0000","last_undegraded":"2024-03-30T12:01:36.791741+0000","last_fullsized":"2024-03-30T12:01:36.791741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793424+0000","last_change":"2024-03-30T12:01:35.669063+0000","last_active":"2024-03-30T12:01:36.793424+0000","last_peered":"2024-03-30T12:01:36.793424+0000","last_clean":"2024-03-30T12:01:36.793424+0000","last_became_active":"2024-03-30T12:01:35.668604+0000","last_became_peered":"2024-03-30T12:01:35.668604+0000","last_unstale":"2024-03-30T12:01:36.793424+0000","last_undegraded":"2024-03-30T12:01:36.793424+0000","last_fullsized":"2024-03-30T12:01:36.793424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.650022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793247+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:36.793247+0000","last_peered":"2024-03-30T12:01:36.793247+0000","last_clean":"2024-03-30T12:01:36.793247+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:36.793247+0000","last_undegraded":"2024-03-30T12:01:36.793247+0000","last_fullsized":"2024-03-30T12:01:36.793247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'105","reported_seq":147,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.843454+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:36.843454+0000","last_peered":"2024-03-30T12:01:36.843454+0000","last_clean":"2024-03-30T12:01:36.843454+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:36.843454+0000","last_undegraded":"2024-03-30T12:01:36.843454+0000","last_fullsized":"2024-03-30T12:01:36.843454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":934976,"data_compressed":11404,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":41121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":147456,"data_stored":44406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":447197,"data_compressed":3064,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":106496,"data_stored":40637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":151552,"data_stored":45248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27124,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344716,"statfs":{"total":96636764160,"available":96608989184,"internally_reserved":0,"allocated":106496,"data_stored":36288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":147456,"data_stored":445161,"data_compressed":3064,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":151552,"data_stored":45248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":467488,"data_compressed":5702,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":467488,"data_compressed":5702,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:01:39.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:01:39.735 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:39.735 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:01:39.752 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-30T12:01:39.602513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221576,"kb_used_data":1192,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753144,"statfs":{"total":773094113280,"available":772867219456,"internally_reserved":0,"allocated":1220608,"data_stored":1447232,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000609"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793282+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:36.793282+0000","last_peered":"2024-03-30T12:01:36.793282+0000","last_clean":"2024-03-30T12:01:36.793282+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:36.793282+0000","last_undegraded":"2024-03-30T12:01:36.793282+0000","last_fullsized":"2024-03-30T12:01:36.793282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.656496+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:36.656496+0000","last_peered":"2024-03-30T12:01:36.656496+0000","last_clean":"2024-03-30T12:01:36.656496+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:36.656496+0000","last_undegraded":"2024-03-30T12:01:36.656496+0000","last_fullsized":"2024-03-30T12:01:36.656496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793315+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:36.793315+0000","last_peered":"2024-03-30T12:01:36.793315+0000","last_clean":"2024-03-30T12:01:36.793315+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:36.793315+0000","last_undegraded":"2024-03-30T12:01:36.793315+0000","last_fullsized":"2024-03-30T12:01:36.793315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793201+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:36.793201+0000","last_peered":"2024-03-30T12:01:36.793201+0000","last_clean":"2024-03-30T12:01:36.793201+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:36.793201+0000","last_undegraded":"2024-03-30T12:01:36.793201+0000","last_fullsized":"2024-03-30T12:01:36.793201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793242+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:36.793242+0000","last_peered":"2024-03-30T12:01:36.793242+0000","last_clean":"2024-03-30T12:01:36.793242+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:36.793242+0000","last_undegraded":"2024-03-30T12:01:36.793242+0000","last_fullsized":"2024-03-30T12:01:36.793242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.791741+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:36.791741+0000","last_peered":"2024-03-30T12:01:36.791741+0000","last_clean":"2024-03-30T12:01:36.791741+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:36.791741+0000","last_undegraded":"2024-03-30T12:01:36.791741+0000","last_fullsized":"2024-03-30T12:01:36.791741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793424+0000","last_change":"2024-03-30T12:01:35.669063+0000","last_active":"2024-03-30T12:01:36.793424+0000","last_peered":"2024-03-30T12:01:36.793424+0000","last_clean":"2024-03-30T12:01:36.793424+0000","last_became_active":"2024-03-30T12:01:35.668604+0000","last_became_peered":"2024-03-30T12:01:35.668604+0000","last_unstale":"2024-03-30T12:01:36.793424+0000","last_undegraded":"2024-03-30T12:01:36.793424+0000","last_fullsized":"2024-03-30T12:01:36.793424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.650022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793247+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:36.793247+0000","last_peered":"2024-03-30T12:01:36.793247+0000","last_clean":"2024-03-30T12:01:36.793247+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:36.793247+0000","last_undegraded":"2024-03-30T12:01:36.793247+0000","last_fullsized":"2024-03-30T12:01:36.793247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:37.288297+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:37.288297+0000","last_peered":"2024-03-30T12:01:37.288297+0000","last_clean":"2024-03-30T12:01:37.288297+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:37.288297+0000","last_undegraded":"2024-03-30T12:01:37.288297+0000","last_fullsized":"2024-03-30T12:01:37.288297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":126976,"data_stored":47239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":159744,"data_stored":50524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":188416,"data_stored":576727,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":46755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":48564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":167936,"data_stored":574691,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:01:39.753 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T12:01:39.753 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T12:01:39.753 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-30T12:01:39.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-30T12:01:40.051 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:40.051 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T12:01:40.068 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-30T12:01:40.069 INFO:teuthology.run_tasks:Running task exec... 2024-03-30T12:01:40.087 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T12:01:40.087 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:01:40.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/balancer/log_level debug' 2024-03-30T12:01:40.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-03-30T12:01:40.650 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "active": true, 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_duration": "0:00:00.000855", 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_started": "Sat Mar 30 12:01:27 2024", 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "mode": "upmap", 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "no_optimization_needed": false, 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "optimize_result": "No pools available", 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "plans": [] 2024-03-30T12:01:40.651 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:01:40.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client reef' 2024-03-30T12:01:41.692 INFO:teuthology.orchestra.run.smithi172.stderr:set require_min_compat_client to reef 2024-03-30T12:01:41.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode upmap-read' 2024-03-30T12:01:42.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-03-30T12:01:42.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-03-30T12:01:42.593 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "active": true, 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_duration": "0:00:00.001220", 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_started": "Sat Mar 30 12:01:42 2024", 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "mode": "upmap-read", 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "no_optimization_needed": true, 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "optimize_result": "Optimization plan created successfully", 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout: "plans": [] 2024-03-30T12:01:42.594 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:01:42.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status detail' 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "active": true, 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_duration": "0:00:00.001220", 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "last_optimize_started": "Sat Mar 30 12:01:42 2024", 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "mode": "upmap-read", 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "no_optimization_needed": true, 2024-03-30T12:01:42.911 INFO:teuthology.orchestra.run.smithi172.stdout: "optimize_result": "Optimization plan created successfully", 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_items_added": [], 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_items_removed": [], 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_primaries_added": [], 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout: "pg_upmap_primaries_removed": [], 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout: "plans": [] 2024-03-30T12:01:42.912 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:01:42.930 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-30T12:01:42.940 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.thrashosds' 2024-03-30T12:01:43.139 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=eabec34a46eb443b905258072c46fecd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.thrashosds' 2024-03-30T12:01:43.140 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-30T12:01:43.152 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T12:01:43.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:01:43.181 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-30T12:01:43.188+0000 7fe2c3b47640 -1 osd.5 17 handle_osd_map injecting map crc failure 2024-03-30T12:01:43.464 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:43.464 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:01:43.483 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-03-30T12:01:43.015234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221576,"kb_used_data":1192,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753144,"statfs":{"total":773094113280,"available":772867219456,"internally_reserved":0,"allocated":1220608,"data_stored":1447232,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.413330"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793282+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:36.793282+0000","last_peered":"2024-03-30T12:01:36.793282+0000","last_clean":"2024-03-30T12:01:36.793282+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:36.793282+0000","last_undegraded":"2024-03-30T12:01:36.793282+0000","last_fullsized":"2024-03-30T12:01:36.793282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.656496+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:36.656496+0000","last_peered":"2024-03-30T12:01:36.656496+0000","last_clean":"2024-03-30T12:01:36.656496+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:36.656496+0000","last_undegraded":"2024-03-30T12:01:36.656496+0000","last_fullsized":"2024-03-30T12:01:36.656496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793315+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:36.793315+0000","last_peered":"2024-03-30T12:01:36.793315+0000","last_clean":"2024-03-30T12:01:36.793315+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:36.793315+0000","last_undegraded":"2024-03-30T12:01:36.793315+0000","last_fullsized":"2024-03-30T12:01:36.793315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793201+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:36.793201+0000","last_peered":"2024-03-30T12:01:36.793201+0000","last_clean":"2024-03-30T12:01:36.793201+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:36.793201+0000","last_undegraded":"2024-03-30T12:01:36.793201+0000","last_fullsized":"2024-03-30T12:01:36.793201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793242+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:36.793242+0000","last_peered":"2024-03-30T12:01:36.793242+0000","last_clean":"2024-03-30T12:01:36.793242+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:36.793242+0000","last_undegraded":"2024-03-30T12:01:36.793242+0000","last_fullsized":"2024-03-30T12:01:36.793242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.791741+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:36.791741+0000","last_peered":"2024-03-30T12:01:36.791741+0000","last_clean":"2024-03-30T12:01:36.791741+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:36.791741+0000","last_undegraded":"2024-03-30T12:01:36.791741+0000","last_fullsized":"2024-03-30T12:01:36.791741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793424+0000","last_change":"2024-03-30T12:01:35.669063+0000","last_active":"2024-03-30T12:01:36.793424+0000","last_peered":"2024-03-30T12:01:36.793424+0000","last_clean":"2024-03-30T12:01:36.793424+0000","last_became_active":"2024-03-30T12:01:35.668604+0000","last_became_peered":"2024-03-30T12:01:35.668604+0000","last_unstale":"2024-03-30T12:01:36.793424+0000","last_undegraded":"2024-03-30T12:01:36.793424+0000","last_fullsized":"2024-03-30T12:01:36.793424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.650022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793247+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:36.793247+0000","last_peered":"2024-03-30T12:01:36.793247+0000","last_clean":"2024-03-30T12:01:36.793247+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:36.793247+0000","last_undegraded":"2024-03-30T12:01:36.793247+0000","last_fullsized":"2024-03-30T12:01:36.793247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:37.288297+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:37.288297+0000","last_peered":"2024-03-30T12:01:37.288297+0000","last_clean":"2024-03-30T12:01:37.288297+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:37.288297+0000","last_undegraded":"2024-03-30T12:01:37.288297+0000","last_fullsized":"2024-03-30T12:01:37.288297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":126976,"data_stored":47239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":159744,"data_stored":50524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":188416,"data_stored":576727,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":46755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":48564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":167936,"data_stored":574691,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:01:43.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:01:43.770 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:43.770 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:01:43.787 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-30T12:01:43.603664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221576,"kb_used_data":1192,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753144,"statfs":{"total":773094113280,"available":772867219456,"internally_reserved":0,"allocated":1220608,"data_stored":1447232,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001760"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793282+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:36.793282+0000","last_peered":"2024-03-30T12:01:36.793282+0000","last_clean":"2024-03-30T12:01:36.793282+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:36.793282+0000","last_undegraded":"2024-03-30T12:01:36.793282+0000","last_fullsized":"2024-03-30T12:01:36.793282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.656496+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:36.656496+0000","last_peered":"2024-03-30T12:01:36.656496+0000","last_clean":"2024-03-30T12:01:36.656496+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:36.656496+0000","last_undegraded":"2024-03-30T12:01:36.656496+0000","last_fullsized":"2024-03-30T12:01:36.656496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793315+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:36.793315+0000","last_peered":"2024-03-30T12:01:36.793315+0000","last_clean":"2024-03-30T12:01:36.793315+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:36.793315+0000","last_undegraded":"2024-03-30T12:01:36.793315+0000","last_fullsized":"2024-03-30T12:01:36.793315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793201+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:36.793201+0000","last_peered":"2024-03-30T12:01:36.793201+0000","last_clean":"2024-03-30T12:01:36.793201+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:36.793201+0000","last_undegraded":"2024-03-30T12:01:36.793201+0000","last_fullsized":"2024-03-30T12:01:36.793201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793242+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:36.793242+0000","last_peered":"2024-03-30T12:01:36.793242+0000","last_clean":"2024-03-30T12:01:36.793242+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:36.793242+0000","last_undegraded":"2024-03-30T12:01:36.793242+0000","last_fullsized":"2024-03-30T12:01:36.793242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.791741+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:36.791741+0000","last_peered":"2024-03-30T12:01:36.791741+0000","last_clean":"2024-03-30T12:01:36.791741+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:36.791741+0000","last_undegraded":"2024-03-30T12:01:36.791741+0000","last_fullsized":"2024-03-30T12:01:36.791741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793424+0000","last_change":"2024-03-30T12:01:35.669063+0000","last_active":"2024-03-30T12:01:36.793424+0000","last_peered":"2024-03-30T12:01:36.793424+0000","last_clean":"2024-03-30T12:01:36.793424+0000","last_became_active":"2024-03-30T12:01:35.668604+0000","last_became_peered":"2024-03-30T12:01:35.668604+0000","last_unstale":"2024-03-30T12:01:36.793424+0000","last_undegraded":"2024-03-30T12:01:36.793424+0000","last_fullsized":"2024-03-30T12:01:36.793424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.650022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:36.793247+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:36.793247+0000","last_peered":"2024-03-30T12:01:36.793247+0000","last_clean":"2024-03-30T12:01:36.793247+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:36.793247+0000","last_undegraded":"2024-03-30T12:01:36.793247+0000","last_fullsized":"2024-03-30T12:01:36.793247+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":162,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-30T12:01:37.288297+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:37.288297+0000","last_peered":"2024-03-30T12:01:37.288297+0000","last_clean":"2024-03-30T12:01:37.288297+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:37.288297+0000","last_undegraded":"2024-03-30T12:01:37.288297+0000","last_fullsized":"2024-03-30T12:01:37.288297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":126976,"data_stored":47239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31340,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":159744,"data_stored":50524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":188416,"data_stored":576727,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":118784,"data_stored":46755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":48564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":167936,"data_stored":574691,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":51366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:01:43.788 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T12:01:43.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:01:44.057 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:44.058 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:01:44.075 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-30T12:01:44.023102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221720,"kb_used_data":1360,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753000,"statfs":{"total":773094113280,"available":772867072000,"internally_reserved":0,"allocated":1392640,"data_stored":1523266,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.421198"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.193775+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:43.193775+0000","last_peered":"2024-03-30T12:01:43.193775+0000","last_clean":"2024-03-30T12:01:43.193775+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:43.193775+0000","last_undegraded":"2024-03-30T12:01:43.193775+0000","last_fullsized":"2024-03-30T12:01:43.193775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.193578+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:43.193578+0000","last_peered":"2024-03-30T12:01:43.193578+0000","last_clean":"2024-03-30T12:01:43.193578+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:43.193578+0000","last_undegraded":"2024-03-30T12:01:43.193578+0000","last_fullsized":"2024-03-30T12:01:43.193578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.193618+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:43.193618+0000","last_peered":"2024-03-30T12:01:43.193618+0000","last_clean":"2024-03-30T12:01:43.193618+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:43.193618+0000","last_undegraded":"2024-03-30T12:01:43.193618+0000","last_fullsized":"2024-03-30T12:01:43.193618+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.197945+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:43.197945+0000","last_peered":"2024-03-30T12:01:43.197945+0000","last_clean":"2024-03-30T12:01:43.197945+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:43.197945+0000","last_undegraded":"2024-03-30T12:01:43.197945+0000","last_fullsized":"2024-03-30T12:01:43.197945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.193574+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:43.193574+0000","last_peered":"2024-03-30T12:01:43.193574+0000","last_clean":"2024-03-30T12:01:43.193574+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:43.193574+0000","last_undegraded":"2024-03-30T12:01:43.193574+0000","last_fullsized":"2024-03-30T12:01:43.193574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.020263+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:43.020263+0000","last_peered":"2024-03-30T12:01:43.020263+0000","last_clean":"2024-03-30T12:01:43.020263+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:43.020263+0000","last_undegraded":"2024-03-30T12:01:43.020263+0000","last_fullsized":"2024-03-30T12:01:43.020263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3,"reported_epoch":19,"state":"peering","last_fresh":"2024-03-30T12:01:43.198530+0000","last_change":"2024-03-30T12:01:43.198530+0000","last_active":"2024-03-30T12:01:34.640338+0000","last_peered":"2024-03-30T12:01:34.640338+0000","last_clean":"2024-03-30T12:01:34.640338+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-30T12:01:43.198530+0000","last_undegraded":"2024-03-30T12:01:43.198530+0000","last_fullsized":"2024-03-30T12:01:43.198530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.193769+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:43.193769+0000","last_peered":"2024-03-30T12:01:43.193769+0000","last_clean":"2024-03-30T12:01:43.193769+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:43.193769+0000","last_undegraded":"2024-03-30T12:01:43.193769+0000","last_fullsized":"2024-03-30T12:01:43.193769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-30T12:01:43.194284+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:43.194284+0000","last_peered":"2024-03-30T12:01:43.194284+0000","last_clean":"2024-03-30T12:01:43.194284+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:43.194284+0000","last_undegraded":"2024-03-30T12:01:43.194284+0000","last_fullsized":"2024-03-30T12:01:43.194284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":151552,"data_stored":58101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31364,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340476,"statfs":{"total":96636764160,"available":96604647424,"internally_reserved":0,"allocated":184320,"data_stored":61386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":587589,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":57617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":59426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":167936,"data_stored":574691,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":188416,"data_stored":62228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:01:44.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:01:44.343 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:01:44.343 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":20,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:44.015150+0000","last_up_change":"2024-03-30T12:01:32.619553+0000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T12:01:33.686010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T12:01:34.619044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6801","nonce":2789107731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6803","nonce":2789107731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6807","nonce":2789107731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6805","nonce":2789107731}]},"public_addr":"172.21.15.172:6801/2789107731","cluster_addr":"172.21.15.172:6803/2789107731","heartbeat_back_addr":"172.21.15.172:6807/2789107731","heartbeat_front_addr":"172.21.15.172:6805/2789107731","state":["exists","up"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6809","nonce":307083902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6811","nonce":307083902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6815","nonce":307083902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6813","nonce":307083902}]},"public_addr":"172.21.15.172:6809/307083902","cluster_addr":"172.21.15.172:6811/307083902","heartbeat_back_addr":"172.21.15.172:6815/307083902","heartbeat_front_addr":"172.21.15.172:6813/307083902","state":["exists","up"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6819","nonce":3212069946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6821","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6822","nonce":3212069946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6831","nonce":3212069946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6826","nonce":3212069946}]},"public_addr":"172.21.15.172:6819/3212069946","cluster_addr":"172.21.15.172:6822/3212069946","heartbeat_back_addr":"172.21.15.172:6831/3212069946","heartbeat_front_addr":"172.21.15.172:6826/3212069946","state":["exists","up"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6818","nonce":1025508895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6823","nonce":1025508895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6829","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6830","nonce":1025508895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6825","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6827","nonce":1025508895}]},"public_addr":"172.21.15.172:6818/1025508895","cluster_addr":"172.21.15.172:6823/1025508895","heartbeat_back_addr":"172.21.15.172:6830/1025508895","heartbeat_front_addr":"172.21.15.172:6827/1025508895","state":["exists","up"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6809","nonce":3222045438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6811","nonce":3222045438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6817","nonce":3222045438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6813","nonce":3222045438}]},"public_addr":"172.21.15.178:6809/3222045438","cluster_addr":"172.21.15.178:6811/3222045438","heartbeat_back_addr":"172.21.15.178:6817/3222045438","heartbeat_front_addr":"172.21.15.178:6813/3222045438","state":["exists","up"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6816","nonce":3461836880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6819","nonce":3461836880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6823","nonce":3461836880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6821","nonce":3461836880}]},"public_addr":"172.21.15.178:6816/3461836880","cluster_addr":"172.21.15.178:6819/3461836880","heartbeat_back_addr":"172.21.15.178:6823/3461836880","heartbeat_front_addr":"172.21.15.178:6821/3461836880","state":["exists","up"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6801","nonce":495238561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6803","nonce":495238561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6807","nonce":495238561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6805","nonce":495238561}]},"public_addr":"172.21.15.178:6801/495238561","cluster_addr":"172.21.15.178:6803/495238561","heartbeat_back_addr":"172.21.15.178:6807/495238561","heartbeat_front_addr":"172.21.15.178:6805/495238561","state":["exists","up"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6825","nonce":2824579992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6827","nonce":2824579992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6831","nonce":2824579992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6829","nonce":2824579992}]},"public_addr":"172.21.15.178:6825/2824579992","cluster_addr":"172.21.15.178:6827/2824579992","heartbeat_back_addr":"172.21.15.178:6831/2824579992","heartbeat_front_addr":"172.21.15.178:6829/2824579992","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.353563+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.465604+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.409253+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.429436+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.518286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.469883+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.366013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.437737+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[{"pgid":"2.0","primary_osd":5}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:01:44.362 INFO:tasks.ceph:Waiting for all PGs to be active+clean and split+merged, waiting on ['2.0'] to go clean 2024-03-30T12:02:04.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:02:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:02:04.665 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:02:04.683 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-30T12:02:03.609835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221820,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752900,"statfs":{"total":773094113280,"available":772866969600,"internally_reserved":0,"allocated":1515520,"data_stored":1577640,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003887"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028859+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:44.028859+0000","last_peered":"2024-03-30T12:01:44.028859+0000","last_clean":"2024-03-30T12:01:44.028859+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:44.028859+0000","last_undegraded":"2024-03-30T12:01:44.028859+0000","last_fullsized":"2024-03-30T12:01:44.028859+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:49.974988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.025664+0000","last_change":"2024-03-30T12:01:36.065202+0000","last_active":"2024-03-30T12:01:44.025664+0000","last_peered":"2024-03-30T12:01:44.025664+0000","last_clean":"2024-03-30T12:01:44.025664+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:01:44.025664+0000","last_undegraded":"2024-03-30T12:01:44.025664+0000","last_fullsized":"2024-03-30T12:01:44.025664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.965665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028753+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:44.028753+0000","last_peered":"2024-03-30T12:01:44.028753+0000","last_clean":"2024-03-30T12:01:44.028753+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:44.028753+0000","last_undegraded":"2024-03-30T12:01:44.028753+0000","last_fullsized":"2024-03-30T12:01:44.028753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:46.057986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.029838+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:44.029838+0000","last_peered":"2024-03-30T12:01:44.029838+0000","last_clean":"2024-03-30T12:01:44.029838+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:44.029838+0000","last_undegraded":"2024-03-30T12:01:44.029838+0000","last_fullsized":"2024-03-30T12:01:44.029838+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028737+0000","last_change":"2024-03-30T12:01:35.667109+0000","last_active":"2024-03-30T12:01:44.028737+0000","last_peered":"2024-03-30T12:01:44.028737+0000","last_clean":"2024-03-30T12:01:44.028737+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:01:44.028737+0000","last_undegraded":"2024-03-30T12:01:44.028737+0000","last_fullsized":"2024-03-30T12:01:44.028737+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:00.312674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:45.605565+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:45.605565+0000","last_peered":"2024-03-30T12:01:45.605565+0000","last_clean":"2024-03-30T12:01:45.605565+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:45.605565+0000","last_undegraded":"2024-03-30T12:01:45.605565+0000","last_fullsized":"2024-03-30T12:01:45.605565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.032939+0000","last_change":"2024-03-30T12:01:44.032939+0000","last_active":"2024-03-30T12:01:44.032939+0000","last_peered":"2024-03-30T12:01:44.032939+0000","last_clean":"2024-03-30T12:01:44.032939+0000","last_became_active":"2024-03-30T12:01:44.032314+0000","last_became_peered":"2024-03-30T12:01:44.032314+0000","last_unstale":"2024-03-30T12:01:44.032939+0000","last_undegraded":"2024-03-30T12:01:44.032939+0000","last_fullsized":"2024-03-30T12:01:44.032939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:40.210147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028745+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:44.028745+0000","last_peered":"2024-03-30T12:01:44.028745+0000","last_clean":"2024-03-30T12:01:44.028745+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:44.028745+0000","last_undegraded":"2024-03-30T12:01:44.028745+0000","last_fullsized":"2024-03-30T12:01:44.028745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:45.467691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:45.605896+0000","last_change":"2024-03-30T12:01:35.669175+0000","last_active":"2024-03-30T12:01:45.605896+0000","last_peered":"2024-03-30T12:01:45.605896+0000","last_clean":"2024-03-30T12:01:45.605896+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:01:45.605896+0000","last_undegraded":"2024-03-30T12:01:45.605896+0000","last_fullsized":"2024-03-30T12:01:45.605896+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:59.792702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":63540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":66825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":593028,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":63056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":64865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":590992,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:02:04.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:02:04.981 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:02:04.982 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":20,"fsid":"0958e16d-2906-41ba-a5c4-32c1d269fa2a","created":"2024-03-30T12:01:22.910202+0000","modified":"2024-03-30T12:01:44.015150+0000","last_up_change":"2024-03-30T12:01:32.619553+0000","last_in_change":"2024-03-30T12:01:25.879162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T12:01:33.686010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-30T12:01:34.619044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"076ecae5-da19-444a-bd06-667256ac89ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6801","nonce":2789107731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6803","nonce":2789107731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6807","nonce":2789107731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2789107731},{"type":"v1","addr":"172.21.15.172:6805","nonce":2789107731}]},"public_addr":"172.21.15.172:6801/2789107731","cluster_addr":"172.21.15.172:6803/2789107731","heartbeat_back_addr":"172.21.15.172:6807/2789107731","heartbeat_front_addr":"172.21.15.172:6805/2789107731","state":["exists","up"]},{"osd":1,"uuid":"0ad37a2f-4ebf-4265-8003-f35267f1d7f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6809","nonce":307083902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6811","nonce":307083902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6815","nonce":307083902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":307083902},{"type":"v1","addr":"172.21.15.172:6813","nonce":307083902}]},"public_addr":"172.21.15.172:6809/307083902","cluster_addr":"172.21.15.172:6811/307083902","heartbeat_back_addr":"172.21.15.172:6815/307083902","heartbeat_front_addr":"172.21.15.172:6813/307083902","state":["exists","up"]},{"osd":2,"uuid":"fad1e8ec-c235-45cc-8d4a-40d567b40e74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6819","nonce":3212069946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6821","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6822","nonce":3212069946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6831","nonce":3212069946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3212069946},{"type":"v1","addr":"172.21.15.172:6826","nonce":3212069946}]},"public_addr":"172.21.15.172:6819/3212069946","cluster_addr":"172.21.15.172:6822/3212069946","heartbeat_back_addr":"172.21.15.172:6831/3212069946","heartbeat_front_addr":"172.21.15.172:6826/3212069946","state":["exists","up"]},{"osd":3,"uuid":"24a3b726-2aee-4271-a035-592516efcec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6818","nonce":1025508895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6823","nonce":1025508895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6829","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6830","nonce":1025508895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6825","nonce":1025508895},{"type":"v1","addr":"172.21.15.172:6827","nonce":1025508895}]},"public_addr":"172.21.15.172:6818/1025508895","cluster_addr":"172.21.15.172:6823/1025508895","heartbeat_back_addr":"172.21.15.172:6830/1025508895","heartbeat_front_addr":"172.21.15.172:6827/1025508895","state":["exists","up"]},{"osd":4,"uuid":"9b998b21-6f34-444b-9a0d-c8923b3af8d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6809","nonce":3222045438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6811","nonce":3222045438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6817","nonce":3222045438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":3222045438},{"type":"v1","addr":"172.21.15.178:6813","nonce":3222045438}]},"public_addr":"172.21.15.178:6809/3222045438","cluster_addr":"172.21.15.178:6811/3222045438","heartbeat_back_addr":"172.21.15.178:6817/3222045438","heartbeat_front_addr":"172.21.15.178:6813/3222045438","state":["exists","up"]},{"osd":5,"uuid":"140ae929-bb76-4f4d-a85c-d4c5e8f45562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6816","nonce":3461836880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6819","nonce":3461836880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6823","nonce":3461836880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":3461836880},{"type":"v1","addr":"172.21.15.178:6821","nonce":3461836880}]},"public_addr":"172.21.15.178:6816/3461836880","cluster_addr":"172.21.15.178:6819/3461836880","heartbeat_back_addr":"172.21.15.178:6823/3461836880","heartbeat_front_addr":"172.21.15.178:6821/3461836880","state":["exists","up"]},{"osd":6,"uuid":"7f1634af-d6e6-4f8b-bdb5-b7065f3cf0b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6801","nonce":495238561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6803","nonce":495238561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6807","nonce":495238561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":495238561},{"type":"v1","addr":"172.21.15.178:6805","nonce":495238561}]},"public_addr":"172.21.15.178:6801/495238561","cluster_addr":"172.21.15.178:6803/495238561","heartbeat_back_addr":"172.21.15.178:6807/495238561","heartbeat_front_addr":"172.21.15.178:6805/495238561","state":["exists","up"]},{"osd":7,"uuid":"0a436e85-027e-417c-bb92-3871492fb39c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6825","nonce":2824579992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6827","nonce":2824579992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6831","nonce":2824579992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2824579992},{"type":"v1","addr":"172.21.15.178:6829","nonce":2824579992}]},"public_addr":"172.21.15.178:6825/2824579992","cluster_addr":"172.21.15.178:6827/2824579992","heartbeat_back_addr":"172.21.15.178:6831/2824579992","heartbeat_front_addr":"172.21.15.178:6829/2824579992","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.353563+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.465604+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.409253+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.429436+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.518286+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.469883+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.366013+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:01:31.437737+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[{"pgid":"2.0","primary_osd":5}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:02:06.000 INFO:tasks.ceph:Scrubbing osd.0 2024-03-30T12:02:06.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:06.161 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:06.162 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:06.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-30T12:02:06.453 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 0 to deep-scrub 2024-03-30T12:02:06.470 INFO:tasks.ceph:Scrubbing osd.1 2024-03-30T12:02:06.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:06.618 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:06.618 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:06.618 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:06.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-30T12:02:06.894 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 1 to deep-scrub 2024-03-30T12:02:06.913 INFO:tasks.ceph:Scrubbing osd.2 2024-03-30T12:02:06.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:07.046 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:07.046 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:07.046 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:07.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-30T12:02:07.334 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 2 to deep-scrub 2024-03-30T12:02:07.351 INFO:tasks.ceph:Scrubbing osd.3 2024-03-30T12:02:07.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:07.488 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:07.488 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:07.488 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:07.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-30T12:02:07.764 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 3 to deep-scrub 2024-03-30T12:02:07.781 INFO:tasks.ceph:Scrubbing osd.4 2024-03-30T12:02:07.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:07.921 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:07.922 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:07.922 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:07.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-30T12:02:08.201 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 4 to deep-scrub 2024-03-30T12:02:08.218 INFO:tasks.ceph:Scrubbing osd.5 2024-03-30T12:02:08.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:08.364 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:08.364 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:08.365 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:08.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-30T12:02:08.642 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 5 to deep-scrub 2024-03-30T12:02:08.659 INFO:tasks.ceph:Scrubbing osd.6 2024-03-30T12:02:08.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:08.804 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:08.804 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:08.804 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:08.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-30T12:02:09.083 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 6 to deep-scrub 2024-03-30T12:02:09.100 INFO:tasks.ceph:Scrubbing osd.7 2024-03-30T12:02:09.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-30T12:02:09.238 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-30T12:02:09.239 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T12:02:09.239 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-30T12:02:09.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-30T12:02:09.517 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 7 to deep-scrub 2024-03-30T12:02:09.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:02:09.820 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:02:09.820 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:02:09.836 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-30T12:02:09.611530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221816,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752904,"statfs":{"total":773094113280,"available":772866973696,"internally_reserved":0,"allocated":1515520,"data_stored":1577640,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028972+0000","last_change":"2024-03-30T12:01:36.065906+0000","last_active":"2024-03-30T12:01:44.028972+0000","last_peered":"2024-03-30T12:01:44.028972+0000","last_clean":"2024-03-30T12:01:44.028972+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:01:44.028972+0000","last_undegraded":"2024-03-30T12:01:44.028972+0000","last_fullsized":"2024-03-30T12:01:44.028972+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:08.555105+0000","last_change":"2024-03-30T12:02:08.555158+0000","last_active":"2024-03-30T12:02:08.555105+0000","last_peered":"2024-03-30T12:02:08.555105+0000","last_clean":"2024-03-30T12:02:08.555105+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:02:08.555105+0000","last_undegraded":"2024-03-30T12:02:08.555105+0000","last_fullsized":"2024-03-30T12:02:08.555105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:08.554912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:08.554912+0000","last_clean_scrub_stamp":"2024-03-30T12:02:08.554912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:34.313032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028881+0000","last_change":"2024-03-30T12:01:35.667440+0000","last_active":"2024-03-30T12:01:44.028881+0000","last_peered":"2024-03-30T12:01:44.028881+0000","last_clean":"2024-03-30T12:01:44.028881+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:01:44.028881+0000","last_undegraded":"2024-03-30T12:01:44.028881+0000","last_fullsized":"2024-03-30T12:01:44.028881+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.029838+0000","last_change":"2024-03-30T12:01:35.666188+0000","last_active":"2024-03-30T12:01:44.029838+0000","last_peered":"2024-03-30T12:01:44.029838+0000","last_clean":"2024-03-30T12:01:44.029838+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:01:44.029838+0000","last_undegraded":"2024-03-30T12:01:44.029838+0000","last_fullsized":"2024-03-30T12:01:44.029838+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:35.476032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:08.291793+0000","last_change":"2024-03-30T12:02:08.291848+0000","last_active":"2024-03-30T12:02:08.291793+0000","last_peered":"2024-03-30T12:02:08.291793+0000","last_clean":"2024-03-30T12:02:08.291793+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:02:08.291793+0000","last_undegraded":"2024-03-30T12:02:08.291793+0000","last_fullsized":"2024-03-30T12:02:08.291793+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:08.291606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:08.291606+0000","last_clean_scrub_stamp":"2024-03-30T12:02:08.291606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:12.356721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:45.605565+0000","last_change":"2024-03-30T12:01:35.668425+0000","last_active":"2024-03-30T12:01:45.605565+0000","last_peered":"2024-03-30T12:01:45.605565+0000","last_clean":"2024-03-30T12:01:45.605565+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:01:45.605565+0000","last_undegraded":"2024-03-30T12:01:45.605565+0000","last_fullsized":"2024-03-30T12:01:45.605565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:47.374417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.032939+0000","last_change":"2024-03-30T12:01:44.032939+0000","last_active":"2024-03-30T12:01:44.032939+0000","last_peered":"2024-03-30T12:01:44.032939+0000","last_clean":"2024-03-30T12:01:44.032939+0000","last_became_active":"2024-03-30T12:01:44.032314+0000","last_became_peered":"2024-03-30T12:01:44.032314+0000","last_unstale":"2024-03-30T12:01:44.032939+0000","last_undegraded":"2024-03-30T12:01:44.032939+0000","last_fullsized":"2024-03-30T12:01:44.032939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:02:40.210147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:01:44.028886+0000","last_change":"2024-03-30T12:01:36.382184+0000","last_active":"2024-03-30T12:01:44.028886+0000","last_peered":"2024-03-30T12:01:44.028886+0000","last_clean":"2024-03-30T12:01:44.028886+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:01:44.028886+0000","last_undegraded":"2024-03-30T12:01:44.028886+0000","last_fullsized":"2024-03-30T12:01:44.028886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:01:34.640338+0000","last_clean_scrub_stamp":"2024-03-30T12:01:34.640338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'119","reported_seq":178,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:07.833665+0000","last_change":"2024-03-30T12:02:07.833694+0000","last_active":"2024-03-30T12:02:07.833665+0000","last_peered":"2024-03-30T12:02:07.833665+0000","last_clean":"2024-03-30T12:02:07.833665+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:02:07.833665+0000","last_undegraded":"2024-03-30T12:02:07.833665+0000","last_fullsized":"2024-03-30T12:02:07.833665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'119","last_scrub_stamp":"2024-03-30T12:02:07.833606+0000","last_deep_scrub":"20'119","last_deep_scrub_stamp":"2024-03-30T12:02:07.833606+0000","last_clean_scrub_stamp":"2024-03-30T12:02:07.833606+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:29.919872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":63540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":66825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":593028,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":63056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":64865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":590992,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:02:09.840 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.840 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.840 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.840 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.840 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.841 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-30T12:01:34.640338+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=1, tm_sec=34, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=12, tm_min=2, tm_sec=4, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T12:02:09.841 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-30T12:02:29.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:02:30.149 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-30T12:02:30.150 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-30T12:02:30.164 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-30T12:02:29.616940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221780,"kb_used_data":1480,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752940,"statfs":{"total":773094113280,"available":772867010560,"internally_reserved":0,"allocated":1515520,"data_stored":1577640,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003234"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:10.358704+0000","last_change":"2024-03-30T12:02:10.358804+0000","last_active":"2024-03-30T12:02:10.358704+0000","last_peered":"2024-03-30T12:02:10.358704+0000","last_clean":"2024-03-30T12:02:10.358704+0000","last_became_active":"2024-03-30T12:01:36.065630+0000","last_became_peered":"2024-03-30T12:01:36.065630+0000","last_unstale":"2024-03-30T12:02:10.358704+0000","last_undegraded":"2024-03-30T12:02:10.358704+0000","last_fullsized":"2024-03-30T12:02:10.358704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:10.358440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:10.358440+0000","last_clean_scrub_stamp":"2024-03-30T12:02:10.358440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:39.536409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:08.555105+0000","last_change":"2024-03-30T12:02:08.555158+0000","last_active":"2024-03-30T12:02:08.555105+0000","last_peered":"2024-03-30T12:02:08.555105+0000","last_clean":"2024-03-30T12:02:08.555105+0000","last_became_active":"2024-03-30T12:01:36.064540+0000","last_became_peered":"2024-03-30T12:01:36.064540+0000","last_unstale":"2024-03-30T12:02:08.555105+0000","last_undegraded":"2024-03-30T12:02:08.555105+0000","last_fullsized":"2024-03-30T12:02:08.555105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:08.554912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:08.554912+0000","last_clean_scrub_stamp":"2024-03-30T12:02:08.554912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:34.313032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:09.308067+0000","last_change":"2024-03-30T12:02:09.308087+0000","last_active":"2024-03-30T12:02:09.308067+0000","last_peered":"2024-03-30T12:02:09.308067+0000","last_clean":"2024-03-30T12:02:09.308067+0000","last_became_active":"2024-03-30T12:01:35.666765+0000","last_became_peered":"2024-03-30T12:01:35.666765+0000","last_unstale":"2024-03-30T12:02:09.308067+0000","last_undegraded":"2024-03-30T12:02:09.308067+0000","last_fullsized":"2024-03-30T12:02:09.308067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:09.307985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:09.307985+0000","last_clean_scrub_stamp":"2024-03-30T12:02:09.307985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:30.116137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:09.356649+0000","last_change":"2024-03-30T12:02:09.356704+0000","last_active":"2024-03-30T12:02:09.356649+0000","last_peered":"2024-03-30T12:02:09.356649+0000","last_clean":"2024-03-30T12:02:09.356649+0000","last_became_active":"2024-03-30T12:01:35.665626+0000","last_became_peered":"2024-03-30T12:01:35.665626+0000","last_unstale":"2024-03-30T12:02:09.356649+0000","last_undegraded":"2024-03-30T12:02:09.356649+0000","last_fullsized":"2024-03-30T12:02:09.356649+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:09.356468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:09.356468+0000","last_clean_scrub_stamp":"2024-03-30T12:02:09.356468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:23.341452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:08.291793+0000","last_change":"2024-03-30T12:02:08.291848+0000","last_active":"2024-03-30T12:02:08.291793+0000","last_peered":"2024-03-30T12:02:08.291793+0000","last_clean":"2024-03-30T12:02:08.291793+0000","last_became_active":"2024-03-30T12:01:35.666443+0000","last_became_peered":"2024-03-30T12:01:35.666443+0000","last_unstale":"2024-03-30T12:02:08.291793+0000","last_undegraded":"2024-03-30T12:02:08.291793+0000","last_fullsized":"2024-03-30T12:02:08.291793+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:08.291606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:08.291606+0000","last_clean_scrub_stamp":"2024-03-30T12:02:08.291606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:12.356721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:10.462635+0000","last_change":"2024-03-30T12:02:10.462698+0000","last_active":"2024-03-30T12:02:10.462635+0000","last_peered":"2024-03-30T12:02:10.462635+0000","last_clean":"2024-03-30T12:02:10.462635+0000","last_became_active":"2024-03-30T12:01:35.667598+0000","last_became_peered":"2024-03-30T12:01:35.667598+0000","last_unstale":"2024-03-30T12:02:10.462635+0000","last_undegraded":"2024-03-30T12:02:10.462635+0000","last_fullsized":"2024-03-30T12:02:10.462635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:10.462441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:10.462441+0000","last_clean_scrub_stamp":"2024-03-30T12:02:10.462441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:35.281993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:10.364173+0000","last_change":"2024-03-30T12:02:10.364227+0000","last_active":"2024-03-30T12:02:10.364173+0000","last_peered":"2024-03-30T12:02:10.364173+0000","last_clean":"2024-03-30T12:02:10.364173+0000","last_became_active":"2024-03-30T12:01:44.032314+0000","last_became_peered":"2024-03-30T12:01:44.032314+0000","last_unstale":"2024-03-30T12:02:10.364173+0000","last_undegraded":"2024-03-30T12:02:10.364173+0000","last_fullsized":"2024-03-30T12:02:10.364173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:10.363992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:10.363992+0000","last_clean_scrub_stamp":"2024-03-30T12:02:10.363992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:32.286776+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:11.381685+0000","last_change":"2024-03-30T12:02:11.381742+0000","last_active":"2024-03-30T12:02:11.381685+0000","last_peered":"2024-03-30T12:02:11.381685+0000","last_clean":"2024-03-30T12:02:11.381685+0000","last_became_active":"2024-03-30T12:01:36.381569+0000","last_became_peered":"2024-03-30T12:01:36.381569+0000","last_unstale":"2024-03-30T12:02:11.381685+0000","last_undegraded":"2024-03-30T12:02:11.381685+0000","last_fullsized":"2024-03-30T12:02:11.381685+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:02:11.381494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:02:11.381494+0000","last_clean_scrub_stamp":"2024-03-30T12:02:11.381494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:25.673368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"20'119","reported_seq":178,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-30T12:02:07.833665+0000","last_change":"2024-03-30T12:02:07.833694+0000","last_active":"2024-03-30T12:02:07.833665+0000","last_peered":"2024-03-30T12:02:07.833665+0000","last_clean":"2024-03-30T12:02:07.833665+0000","last_became_active":"2024-03-30T12:01:35.668760+0000","last_became_peered":"2024-03-30T12:01:35.668760+0000","last_unstale":"2024-03-30T12:02:07.833665+0000","last_undegraded":"2024-03-30T12:02:07.833665+0000","last_fullsized":"2024-03-30T12:02:07.833665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'119","last_scrub_stamp":"2024-03-30T12:02:07.833606+0000","last_deep_scrub":"20'119","last_deep_scrub_stamp":"2024-03-30T12:02:07.833606+0000","last_clean_scrub_stamp":"2024-03-30T12:02:07.833606+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:03:29.919872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11610,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":163840,"data_stored":63540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":196608,"data_stored":66825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":593028,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":155648,"data_stored":63056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":64865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":204800,"data_stored":590992,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":200704,"data_stored":67667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5805,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:02:30.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-30T12:02:30.479 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-30T12:02:30.479 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-30T12:02:30.479 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-30T12:02:30.480 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:30.554 INFO:tasks.ceph.osd.0:Stopped 2024-03-30T12:02:30.554 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T12:02:30.554 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:30.651 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T12:02:30.652 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-30T12:02:30.652 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:30.738 INFO:tasks.ceph.osd.2:Stopped 2024-03-30T12:02:30.739 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T12:02:30.739 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:30.836 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T12:02:30.837 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T12:02:30.837 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:30.916 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T12:02:30.916 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T12:02:30.916 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.003 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T12:02:31.004 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T12:02:31.004 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.085 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T12:02:31.086 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T12:02:31.086 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.176 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T12:02:31.176 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-30T12:02:31.177 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-30T12:02:31.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.204 INFO:tasks.ceph.mgr.y:Stopped 2024-03-30T12:02:31.204 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-30T12:02:31.205 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.234 INFO:tasks.ceph.mgr.x:Stopped 2024-03-30T12:02:31.234 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-30T12:02:31.235 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-30T12:02:31.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.268 INFO:tasks.ceph.mon.a:Stopped 2024-03-30T12:02:31.268 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-30T12:02:31.268 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.329 INFO:tasks.ceph.mon.c:Stopped 2024-03-30T12:02:31.330 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-30T12:02:31.330 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:02:31.344 INFO:tasks.ceph.mon.b:Stopped 2024-03-30T12:02:31.345 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-30T12:02:31.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'must scrub before tier agent can activate' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-30T12:02:31.392 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:02:31.392 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-30T12:02:31.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:02:31.695 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-30T12:02:31.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:02:31.912 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-30T12:02:32.093 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:02:32.094 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-30T12:02:32.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:02:32.310 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-30T12:02:32.469 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:02:32.470 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-30T12:02:32.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:02:32.664 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-30T12:02:32.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:02:32.832 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-30T12:02:33.006 INFO:tasks.ceph:Archiving mon data... 2024-03-30T12:02:33.008 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/data/mon.a.tgz 2024-03-30T12:02:33.008 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-30T12:02:33.015 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.ToZn93Yjn7 2024-03-30T12:02:33.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ToZn93Yjn7 2024-03-30T12:02:33.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.ToZn93Yjn7 2024-03-30T12:02:33.174 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.ToZn93Yjn7 is 207KB 2024-03-30T12:02:33.232 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.ToZn93Yjn7 2024-03-30T12:02:33.237 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/data/mon.c.tgz 2024-03-30T12:02:33.238 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-30T12:02:33.286 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.ZyfbNOYORI 2024-03-30T12:02:33.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ZyfbNOYORI 2024-03-30T12:02:33.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.ZyfbNOYORI 2024-03-30T12:02:33.442 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.ZyfbNOYORI is 208KB 2024-03-30T12:02:33.500 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.ZyfbNOYORI 2024-03-30T12:02:33.504 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/data/mon.b.tgz 2024-03-30T12:02:33.504 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-03-30T12:02:33.510 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.LOZ9N8ONrp 2024-03-30T12:02:33.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.LOZ9N8ONrp 2024-03-30T12:02:33.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.LOZ9N8ONrp 2024-03-30T12:02:33.674 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.LOZ9N8ONrp is 208KB 2024-03-30T12:02:33.731 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.LOZ9N8ONrp 2024-03-30T12:02:33.736 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-30T12:02:33.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T12:02:33.739 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T12:02:33.800 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-30T12:02:33.841 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-30T12:02:33.842 INFO:tasks.ceph:Archiving crash dumps... 2024-03-30T12:02:33.843 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi172/crash 2024-03-30T12:02:33.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T12:02:33.858 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi178/crash 2024-03-30T12:02:33.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T12:02:33.897 INFO:tasks.ceph:Compressing logs... 2024-03-30T12:02:33.898 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T12:02:33.903 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T12:02:33.926 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-03-30T12:02:33.927 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30017.log 2024-03-30T12:02:33.927 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-03-30T12:02:33.927 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28944.log 2024-03-30T12:02:33.928 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30017.log.gz 2024-03-30T12:02:33.928 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-30T12:02:33.929 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29705.log 2024-03-30T12:02:33.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28944.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-03-30T12:02:33.929 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28944.log.gz 2024-03-30T12:02:33.930 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25148.log 2024-03-30T12:02:33.930 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29705.log.gz 2024-03-30T12:02:33.930 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28733.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-30T12:02:33.930 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25246.log 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25148.log.gz 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28615.log 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28924.log 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25246.log.gz 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-30T12:02:33.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29166.log 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28615.log.gz 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28744.log 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28924.log.gz 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29166.log.gz 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24988.log 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29861.log 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.24988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24988.log.gz 2024-03-30T12:02:33.935 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29212.log 2024-03-30T12:02:33.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29861.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-30T12:02:33.936 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29861.log.gz 2024-03-30T12:02:33.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29149.log 2024-03-30T12:02:33.936 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-03-30T12:02:33.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29677.log 2024-03-30T12:02:33.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29429.log 2024-03-30T12:02:33.940 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29149.log.gz 2024-03-30T12:02:33.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28647.log 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29677.log.gz 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29429.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29429.log.gz 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28735.log 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28647.log.gz 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30095.log 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28735.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20210.log 2024-03-30T12:02:33.941 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28735.log.gz 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30095.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.28949.log 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30095.log.gz 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.tmp-client.admin.20210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29477.log 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20210.log.gz 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28948.log 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28949.log.gz 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29477.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29887.log 2024-03-30T12:02:33.942 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29477.log.gz 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29835.log 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28948.log.gz 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29965.log/var/log/ceph/ceph-client.admin.29887.log: 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29887.log.gz 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28563.log 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29835.log.gz 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29965.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-30T12:02:33.943 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29965.log.gz 2024-03-30T12:02:33.950 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29651.log 2024-03-30T12:02:33.950 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28563.log.gz 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28688.log 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29651.log.gz 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30223.log 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28688.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28688.log.gz 2024-03-30T12:02:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30223.log.gz 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29991.log 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29213.log 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29599.log 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29991.log.gz 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29213.log.gz 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30043.log 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29599.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-30T12:02:33.952 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29599.log.gz/var/log/ceph/ceph-client.admin.30043.log: 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29573.log 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30043.log.gz 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28731.log 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29573.log: 91.2% -- replaced with /var/log/ceph/ceph.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28538.log 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29573.log.gz 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28731.log.gz 2024-03-30T12:02:33.953 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29625.log 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29913.log 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28538.log.gz 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29525.log 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29625.log.gz 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29913.log.gz 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30147.log 2024-03-30T12:02:33.954 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29525.log.gz 2024-03-30T12:02:33.955 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29501.log 2024-03-30T12:02:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28955.log 2024-03-30T12:02:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30147.log: /var/log/ceph/ceph-client.admin.29501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30147.log.gz 2024-03-30T12:02:33.960 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29501.log.gz 2024-03-30T12:02:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30069.log 2024-03-30T12:02:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29208.log 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28955.log: /var/log/ceph/ceph-client.admin.30069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28955.log.gz 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30069.log.gz 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28589.log 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29208.log: /var/log/ceph/ceph-client.admin.28589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29208.log.gz 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28589.log.gz 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28739.log 2024-03-30T12:02:33.961 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29405.log 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.28739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28739.log.gz 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29145.log 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30173.log 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29405.log: /var/log/ceph/ceph-client.admin.29145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29405.log.gz 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29145.log.gz 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28930.log 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30121.log 2024-03-30T12:02:33.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30173.log: /var/log/ceph/ceph-client.admin.28930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30173.log.gz 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28930.log.gz 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28998.log 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29549.log 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.30121.log: /var/log/ceph/ceph-client.admin.28998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30121.log.gz 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28998.log.gz 2024-03-30T12:02:33.963 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30199.log 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28933.log 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29549.log: /var/log/ceph/ceph-client.admin.30199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29549.log.gz 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30199.log.gz 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29353.log 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29757.log 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28933.log: /var/log/ceph/ceph-client.admin.29353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28933.log.gz 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29353.log.gz 2024-03-30T12:02:33.969 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29783.log 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29757.log: /var/log/ceph/ceph-client.admin.29783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29757.log.gz 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29783.log.gz 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29167.log 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28740.log 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28742.log: /var/log/ceph/ceph-client.admin.29167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29167.log.gz 2024-03-30T12:02:33.970 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29939.log 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28740.log: /var/log/ceph/ceph-client.admin.29731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28740.log.gz 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28745.log 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29939.log: /var/log/ceph/ceph-client.admin.28745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29939.log.gz 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28745.log.gz 2024-03-30T12:02:33.971 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-03-30T12:02:33.972 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29379.log: 0.0%/var/log/ceph/ceph-client.admin.29809.log: -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-03-30T12:02:33.972 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-03-30T12:02:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25128.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25128.log.gz 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25056.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25032.log 2024-03-30T12:02:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25056.log.gz 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25032.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25032.log.gz 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25008.log 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.25008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25104.log 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25008.log.gz 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-30T12:02:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5/var/log/ceph/ceph.audit.log: --verbose -- /var/log/ceph/ceph-client.admin.25183.log 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25159.log 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25104.log.gz 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25183.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25183.log.gz 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-client.admin.25159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25159.log.gz 2024-03-30T12:02:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25080.log 2024-03-30T12:02:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.25080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25080.log.gz 2024-03-30T12:02:33.986 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-30T12:02:33.996 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-30T12:02:33.998 INFO:teuthology.orchestra.run.smithi178.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-30T12:02:34.014 INFO:teuthology.orchestra.run.smithi178.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-30T12:02:34.026 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-30T12:02:34.027 INFO:teuthology.orchestra.run.smithi178.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-30T12:02:34.029 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-30T12:02:34.039 INFO:teuthology.orchestra.run.smithi178.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-30T12:02:34.040 INFO:teuthology.orchestra.run.smithi178.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-30T12:02:34.066 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-30T12:02:34.379 INFO:teuthology.orchestra.run.smithi172.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-30T12:02:34.380 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-30T12:02:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m0.475s 2024-03-30T12:02:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m0.890s 2024-03-30T12:02:34.380 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.064s 2024-03-30T12:02:34.450 INFO:teuthology.orchestra.run.smithi178.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-30T12:02:34.452 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-30T12:02:34.452 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m0.504s 2024-03-30T12:02:34.452 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m0.738s 2024-03-30T12:02:34.452 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.050s 2024-03-30T12:02:34.453 INFO:tasks.ceph:Archiving logs... 2024-03-30T12:02:34.453 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi172/log 2024-03-30T12:02:34.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T12:02:34.629 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi178/log 2024-03-30T12:02:34.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T12:02:34.743 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T12:02:34.754 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T12:02:34.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T12:02:34.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T12:02:34.825 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T12:02:34.835 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T12:02:34.835 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T12:02:34.838 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T12:02:34.850 INFO:teuthology.orchestra.run.smithi178.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================== 2024-03-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi178.stdout:*hv01.front.sepi 44.4.53.2 2 u 27 64 17 0.074 -1.240 0.923 2024-03-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi178.stdout: hv02.front.sepi 63.231.80.2 3 u 25 64 17 0.141 -0.860 1.329 2024-03-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi178.stdout: hv03.front.sepi 74.6.168.72 3 u 21 64 17 0.123 -1.775 1.204 2024-03-30T12:02:34.851 INFO:teuthology.orchestra.run.smithi178.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:02:34.852 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-03-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi172.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 64 17 0.146 -2.376 1.736 2024-03-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 63.231.80.2 3 u 28 64 17 0.089 -1.931 1.638 2024-03-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 30 64 17 0.105 -2.969 1.656 2024-03-30T12:02:34.853 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:02:34.853 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T12:02:34.863 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T12:02:34.864 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T12:02:34.873 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T12:02:34.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T12:02:34.894 INFO:teuthology.task.internal:Duration was 653.747389 seconds 2024-03-30T12:02:34.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T12:02:34.904 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T12:02:34.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T12:02:34.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T12:02:34.943 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T12:02:34.944 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-30T12:02:34.944 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T12:02:34.995 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-03-30T12:02:34.996 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T12:02:35.006 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T12:02:35.006 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T12:02:35.039 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T12:02:35.073 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T12:02:35.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T12:02:35.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T12:02:35.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T12:02:35.179 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T12:02:35.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T12:02:35.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T12:02:35.203 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T12:02:35.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:02:35.236 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:02:35.244 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-30T12:02:35.253 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-03-30T12:02:35.270 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:02:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:02:35.318 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:02:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:02:35.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T12:02:35.336 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T12:02:35.336 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi172 2024-03-30T12:02:35.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T12:02:35.382 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612/remote/smithi178 2024-03-30T12:02:35.383 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T12:02:35.404 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T12:02:35.404 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T12:02:35.423 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T12:02:35.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T12:02:35.466 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T12:02:35.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T12:02:35.480 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T12:02:35.481 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T12:02:35.483 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T12:02:35.487 INFO:teuthology.orchestra.run.smithi172.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 12:02 /home/ubuntu/cephtest 2024-03-30T12:02:35.501 INFO:teuthology.orchestra.run.smithi178.stdout: 658466 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 12:02 /home/ubuntu/cephtest 2024-03-30T12:02:35.503 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T12:02:35.513 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T12:02:35.541 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T12:02:35.564 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-30T12:02:35.564 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T12:02:35.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 11:42:06.869316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDPFboCyO3bM9cLXOTIqjPA3FxZk5ahQ8MGtbCVb0ySQ79yoypJc3uYDzYtn5h4bDP1IlOWFsHQn5j7M2CKKkts='} 2024-03-30T12:02:35.590 DEBUG:teuthology.nuke:shortname: smithi178 2024-03-30T12:02:35.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T12:02:35.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 11:42:06.870627', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPP32OwpGxEByXBvXQdTzC8LSFKogGaZO56cXXA/etDCnz6VFskXcEmmXaqaKJT5/21SSm5rA5Ikq1zBoq5ODBw='} 2024-03-30T12:02:35.648 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-30T12:02:35.648 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T12:02:35.671 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-30T12:02:35.671 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:35.689 INFO:teuthology.orchestra.console.smithi178:Power off 2024-03-30T12:02:35.689 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T12:02:35.700 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-30T12:02:35.713 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-03-30T12:02:35.713 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:35.739 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-30T12:02:39.701 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:39.740 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:39.812 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-30T12:02:39.850 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-30T12:02:43.815 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:43.853 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:43.925 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-30T12:02:43.964 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-03-30T12:02:44.065 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-03-30T12:02:47.927 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:02:48.037 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-30T12:02:48.138 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-30T12:02:48.245 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} duration: 653.7473886013031 failure_reason: No module named 'tasks.thrashosds' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=eabec34a46eb443b905258072c46fecd status: fail success: false 2024-03-30T12:02:48.246 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T12:02:48.330 INFO:teuthology.run:FAIL