2024-09-13T11:50:11.583 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:50:11.607 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:50:11.711 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463 branch: wip-aclamk-cbt-combined description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small} email: akupczyk@redhat.com first_in_suite: false job_id: '7903463' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 30 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 5 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - must scrub before tier agent can activate - \(POOL_APP_NOT_ENABLED\) sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFP/DjLr7LjZFoEx3rZfFyWy/KEE8z8Cpl8G75Onpjj/KCHef4+sE3K/CRmkBYIg/xQhuHJRLjZ/m0bIKZwyMZ0= smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBayXWhE28aRG7QgefN9CMCdB16Da3RkecGxxr5dV5Ui1QSDDmSRbJ+Noo7DPdf6XjnRrDZmUseXXUP49i8Q9o= smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO66PEVIhtIuAaOuMGEaeoD5Jh/5mzNtsSx1/090Wcfk2ugcI5EeHxznnsIVx4Nf54OdPRkozmXohddewD9haMs= smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAb2+9Jxq9B/Ij8scRZgoWoNMe3Rcuy8lJ3ynQnma8E5ZWqbIo8jMXmZ8Fm4y/LuTCmfvLezngNp8HRxWvfqJ24= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 map_discontinuity_sleep_time: 200 min_in: 2 timeout: 1800 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 60 - sudo ceph osd pool set cache target_max_objects 250 - sudo ceph osd pool set cache min_read_recency_for_promote 2 - sudo ceph osd pool set cache min_write_recency_for_promote 2 - rados: clients: - client.0 objects: 500 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:50:11.713 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:50:11.714 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:50:11.714 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:50:11.716 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:50:11.743 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:50:11.744 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:50:11.744 INFO:teuthology.packaging:ref: None 2024-09-13T11:50:11.744 INFO:teuthology.packaging:tag: None 2024-09-13T11:50:11.744 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:50:11.744 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:50:11.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:50:11.900 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T11:50:11.901 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:50:11.908 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:50:11.908 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:50:11.971 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:50:11.990 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:50:11.997 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:50:12.033 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:44:14.569060', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFP/DjLr7LjZFoEx3rZfFyWy/KEE8z8Cpl8G75Onpjj/KCHef4+sE3K/CRmkBYIg/xQhuHJRLjZ/m0bIKZwyMZ0='} 2024-09-13T11:50:12.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:44:14.566164', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBayXWhE28aRG7QgefN9CMCdB16Da3RkecGxxr5dV5Ui1QSDDmSRbJ+Noo7DPdf6XjnRrDZmUseXXUP49i8Q9o='} 2024-09-13T11:50:12.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:44:14.567710', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO66PEVIhtIuAaOuMGEaeoD5Jh/5mzNtsSx1/090Wcfk2ugcI5EeHxznnsIVx4Nf54OdPRkozmXohddewD9haMs='} 2024-09-13T11:50:12.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:44:14.570378', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAb2+9Jxq9B/Ij8scRZgoWoNMe3Rcuy8lJ3ynQnma8E5ZWqbIo8jMXmZ8Fm4y/LuTCmfvLezngNp8HRxWvfqJ24='} 2024-09-13T11:50:12.133 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:50:12.139 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-13T11:50:12.139 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-13T11:50:12.139 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-13T11:50:12.139 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-13T11:50:12.139 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:50:12.307 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f529fa4bbe0>, signals=[15]) 2024-09-13T11:50:12.308 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:50:12.313 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:50:12.313 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T11:50:12.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:50:12.408 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T11:50:12.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:50:12.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T11:50:12.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:50:12.584 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T11:50:12.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:50:12.675 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:50:12.681 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-09-13T11:50:12.700 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-09-13T11:50:12.700 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-09-13T11:50:12.757 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:50:12.758 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:50:12.759 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:50:12.759 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-09-13T11:50:12.786 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-09-13T11:50:12.805 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-09-13T11:50:12.805 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-09-13T11:50:12.863 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-09-13T11:50:12.863 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-09-13T11:50:12.863 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:50:12.864 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:50:12.865 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-09-13T11:50:12.897 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-13T11:50:12.915 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-13T11:50:12.916 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-13T11:50:12.973 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:50:12.974 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:50:12.975 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:50:12.975 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-13T11:50:13.003 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-09-13T11:50:13.022 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-09-13T11:50:13.022 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-09-13T11:50:13.079 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:50:13.080 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:50:13.081 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-09-13T11:50:13.112 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:50:13.123 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:50:13.128 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:50:13.128 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:50:13.132 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:50:13.134 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:50:13.137 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:50:13.153 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:50:13.159 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:50:13.160 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:50:13.188 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:50:13.191 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:50:13.194 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:50:13.220 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:50:13.349 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:50:13.363 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:50:13.363 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:50:13.364 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:50:13.364 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:50:13.370 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:50:13.370 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:50:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:50:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:50:13.391 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-13T11:50:13.391 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-13T11:50:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:50:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:50:13.393 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T11:50:13.394 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-09-13T11:50:13.449 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:50:13.449 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.14.0-503.el9.x86_64 2024-09-13T11:50:13.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2024-09-13T11:50:13.450 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:50:13.451 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-503.el9.x86_64 2024-09-13T11:50:13.451 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-09-13T11:50:13.452 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:50:13.452 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:50:13.453 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-503.el9.x86_64 2024-09-13T11:50:13.453 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-09-13T11:50:13.453 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-503.el9.x86_64 2024-09-13T11:50:13.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-09-13T11:50:15.868 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T11:50:15.946 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-13T11:50:16.162 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-09-13T11:50:16.982 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.2 MB 00:01 2024-09-13T11:50:17.938 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-13T11:50:17.996 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-13T11:50:18.708 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-09-13T11:50:22.814 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.5 MB 00:00 2024-09-13T11:50:23.484 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.5 MB 00:00 2024-09-13T11:50:23.721 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-13T11:50:24.415 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-13T11:50:25.059 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:08 2024-09-13T11:50:25.147 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 19 kB 00:00 2024-09-13T11:50:25.326 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-13T11:50:25.493 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-13T11:50:26.109 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-13T11:50:26.247 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-13T11:50:29.277 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-13T11:50:30.744 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-13T11:50:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-13T11:50:31.624 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-13T11:50:31.757 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T11:50:31.903 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T11:50:33.273 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:50:33.273 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:50:33.373 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:50:33.385 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 79 M 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 126 M 2024-09-13T11:50:33.386 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:50:33.584 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 362 kB/s | 23 kB 00:00 2024-09-13T11:50:33.796 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:50:33.796 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:50:33.883 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:50:33.887 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-09-13T11:50:33.888 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:50:33.889 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 79 M 2024-09-13T11:50:33.889 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 126 M 2024-09-13T11:50:33.889 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:50:33.993 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:50:33.993 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:50:34.090 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:50:34.091 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 324 kB/s | 23 kB 00:00 2024-09-13T11:50:34.109 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:50:34.110 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:50:34.111 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-09-13T11:50:34.111 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:50:34.111 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 79 M 2024-09-13T11:50:34.111 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 126 M 2024-09-13T11:50:34.111 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:50:34.271 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 207 kB/s | 23 kB 00:00 2024-09-13T11:50:34.793 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-13T11:50:35.050 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-13T11:50:35.201 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 9.3 MB/s | 16 MB 00:01 2024-09-13T11:50:35.910 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-09-13T11:50:35.910 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:35.910 INFO:teuthology.orchestra.run.smithi005.stdout:Total 31 MB/s | 79 MB 00:02 2024-09-13T11:50:36.091 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 14 MB/s | 29 MB 00:01 2024-09-13T11:50:36.517 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-09-13T11:50:36.517 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:36.517 INFO:teuthology.orchestra.run.smithi039.stdout:Total 30 MB/s | 79 MB 00:02 2024-09-13T11:50:36.619 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:50:37.060 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:50:37.060 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:50:37.062 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T11:50:37.227 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:50:37.627 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:50:37.627 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:50:37.971 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.1 MB/s | 16 MB 00:03 2024-09-13T11:50:38.050 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:50:38.050 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:50:38.631 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:50:38.631 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:50:38.788 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.3 MB/s | 29 MB 00:04 2024-09-13T11:50:39.027 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:50:39.027 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:50:39.127 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:50:39.138 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:50:39.139 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-09-13T11:50:39.139 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:50:39.139 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 79 M 2024-09-13T11:50:39.139 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 126 M 2024-09-13T11:50:39.139 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:50:39.263 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.8 MB/s | 35 MB 00:05 2024-09-13T11:50:39.264 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:39.264 INFO:teuthology.orchestra.run.smithi037.stdout:Total 15 MB/s | 79 MB 00:05 2024-09-13T11:50:39.289 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 178 kB/s | 23 kB 00:00 2024-09-13T11:50:39.973 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:50:40.384 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:50:40.384 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:50:40.618 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:50:40.923 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 8.9 MB/s | 16 MB 00:01 2024-09-13T11:50:41.221 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:50:41.295 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:50:41.393 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:41.412 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:50:41.412 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:50:41.464 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-09-13T11:50:41.876 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:50:41.967 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:42.023 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 10 MB/s | 29 MB 00:02 2024-09-13T11:50:42.023 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:50:42.024 INFO:teuthology.orchestra.run.smithi087.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-13T11:50:42.710 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:50:43.068 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:50:43.069 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:50:43.997 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:50:44.033 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:50:44.033 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:50:44.192 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:44.313 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:44.689 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:50:44.761 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:44.772 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:44.889 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:46.454 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:50:47.166 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:50:47.259 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:47.578 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:47.701 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:49.443 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:49.480 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:50.015 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:50.056 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:50.066 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:50:50.203 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:52.827 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:52.868 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:54.622 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:55.061 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:50:55.186 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:55.188 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:50:58.004 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:00.009 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.035 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.069 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.075 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.110 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.389 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.389 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:51:34.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:51:34.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:51:34.409 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:34.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:51:34.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:51:34.409 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:51:35.206 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:35.206 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:51:35.206 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:51:35.207 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:51:35.272 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:51:35.273 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:51:35.543 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:35.634 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:35.639 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.640 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.640 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.640 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:51:35.640 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:51:35.640 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:51:35.640 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:51:35.641 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:35.698 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.699 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-09-13T11:51:35.724 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.725 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.725 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.725 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:51:35.725 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:51:35.725 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-13T11:51:35.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:51:35.726 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:35.730 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-09-13T11:51:35.730 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-09-13T11:51:35.730 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-09-13T11:51:35.730 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 82.el9 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8274670 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-09-13T11:51:35.731 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-09-13T11:51:35.732 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:51:35.733 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:51:35.734 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:51:35.734 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.734 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:51:35.785 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:35.786 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-09-13T11:51:35.824 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-09-13T11:51:35.824 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-09-13T11:51:35.824 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-09-13T11:51:35.824 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 82.el9 2024-09-13T11:51:35.824 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8274670 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:51:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:51:35.826 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:51:35.826 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-09-13T11:51:35.826 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-09-13T11:51:35.826 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:51:35.826 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-09-13T11:51:35.827 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-09-13T11:51:35.827 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:51:35.827 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:51:35.827 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:51:35.828 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:51:35.828 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-09-13T11:51:35.828 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:51:35.828 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:51:35.829 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:51:35.829 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:51:35.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:51:36.374 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-09-13T11:51:36.498 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-13T11:51:36.840 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:36.881 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:37.257 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:37.257 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:51:37.257 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:51:37.258 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:51:37.720 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:51:37.750 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-09-13T11:51:37.761 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T11:51:37.793 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.wEBiwBNPnk 2024-09-13T11:51:37.794 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wEBiwBNPnk 2024-09-13T11:51:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:51:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-13T11:51:37.856 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-13T11:51:37.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.wEBiwBNPnk 2024-09-13T11:51:37.872 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.m0OZXWXY7d 2024-09-13T11:51:37.872 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m0OZXWXY7d 2024-09-13T11:51:37.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.m0OZXWXY7d 2024-09-13T11:51:38.004 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.wEBiwBNPnk is 6KB 2024-09-13T11:51:38.053 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.wEBiwBNPnk 2024-09-13T11:51:38.068 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:51:38.071 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.m0OZXWXY7d is 6KB 2024-09-13T11:51:38.089 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:38.120 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.m0OZXWXY7d 2024-09-13T11:51:38.129 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:51:38.130 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:51:38.130 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:51:38.130 INFO:teuthology.orchestra.run.smithi005.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:51:38.130 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:38.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.134 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.166 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.167 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.167 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:51:38.167 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:51:38.196 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:51:38.197 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:51:38.197 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:51:38.197 INFO:teuthology.orchestra.run.smithi039.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:51:38.198 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-09-13T11:51:38.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-13T11:51:38.426 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-09-13T11:51:38.501 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:38.501 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:51:38.501 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:51:38.501 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:51:38.530 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:38.618 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.619 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:38.619 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:51:38.619 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:51:38.619 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:51:38.619 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T11:51:38.619 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:51:38.619 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:38.679 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:38.679 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:38.679 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 82.el9 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-09-13T11:51:38.720 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8274670 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-09-13T11:51:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:51:38.722 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:51:38.723 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:51:38.723 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:51:38.723 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:51:39.291 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:51:39.412 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-09-13T11:51:39.652 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:39.741 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:39.742 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:51:39.742 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:51:39.742 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:51:39.742 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-13T11:51:39.742 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:51:39.742 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:51:39.800 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:51:39.800 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:51:39.800 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 82.el9 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-09-13T11:51:39.832 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8274670 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:51:39.833 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:51:39.834 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:51:39.834 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:51:39.834 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-09-13T11:51:39.834 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:51:39.834 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:51:39.835 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:51:39.835 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:51:39.835 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:51:40.502 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-09-13T11:51:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:51:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-09-13T11:51:40.712 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-13T11:51:40.727 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.dk3czqyWp2 2024-09-13T11:51:40.728 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dk3czqyWp2 2024-09-13T11:51:40.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.dk3czqyWp2 2024-09-13T11:51:40.927 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.dk3czqyWp2 is 6KB 2024-09-13T11:51:40.975 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.dk3czqyWp2 2024-09-13T11:51:40.990 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:51:41.051 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:51:41.051 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:51:41.052 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:51:41.052 INFO:teuthology.orchestra.run.smithi037.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:51:41.053 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:51:41.178 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-09-13T11:51:41.829 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:51:41.864 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-09-13T11:51:41.866 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-09-13T11:51:41.910 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.7b1qk7s9nt 2024-09-13T11:51:41.910 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7b1qk7s9nt 2024-09-13T11:51:41.977 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.7b1qk7s9nt 2024-09-13T11:51:42.139 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.7b1qk7s9nt is 6KB 2024-09-13T11:51:42.188 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.7b1qk7s9nt 2024-09-13T11:51:42.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:51:42.266 INFO:teuthology.orchestra.run.smithi087.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:51:42.266 INFO:teuthology.orchestra.run.smithi087.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:51:42.266 INFO:teuthology.orchestra.run.smithi087.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:51:42.266 INFO:teuthology.orchestra.run.smithi087.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:51:42.268 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:51:42.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-09-13T11:52:08.309 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:52:08.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:52:08.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:08.347 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-13T11:52:08.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:52:08.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:11.181 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T11:52:11.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:52:11.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:12.426 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-13T11:52:12.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:52:12.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:26.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-09-13T11:52:26.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:52:29.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-13T11:52:30.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-09-13T11:52:35.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:52:35.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:35.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:52:35.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:38.749 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-09-13T11:52:38.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:52:38.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:38.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:52:39.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:52:39.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:42.845 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-09-13T11:52:50.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:52:50.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:50.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:52:50.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:53.857 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:52:54.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:52:54.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:52:57.153 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-09-13T11:53:08.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:53:08.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:53:09.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:53:09.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:53:09.591 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-13T11:53:10.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:53:10.031 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:53:10.031 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-09-13T11:53:10.046 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:53:10.047 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:53:10.047 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:53:10.047 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:53:11.048 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T11:53:11.048 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:53:11.131 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-09-13T11:53:11.157 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:11.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-13T11:53:22.609 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T11:53:22.973 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:53:22.973 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:53:22.973 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-09-13T11:53:22.988 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:53:22.989 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:53:22.989 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:53:22.989 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:53:23.990 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:53:23.990 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:53:24.096 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-09-13T11:53:24.117 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:26.787 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-09-13T11:53:27.185 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:53:27.185 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:53:27.185 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-09-13T11:53:27.200 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:53:27.200 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:53:27.200 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:53:27.200 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:53:28.201 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-13T11:53:28.201 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:53:28.325 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-09-13T11:53:28.345 DEBUG:teuthology.parallel:result is None 2024-09-13T11:53:29.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:53:29.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:54:01.666 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-13T11:54:02.018 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:54:02.018 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:54:02.018 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-13T11:54:02.035 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:54:02.035 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:54:02.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:54:02.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:54:03.037 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-13T11:54:03.037 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:54:03.117 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-13T11:54:03.142 DEBUG:teuthology.parallel:result is None 2024-09-13T11:54:03.142 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:54:03.150 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:54:03.150 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:54:03.154 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:54:03.157 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:54:03.159 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:54:03.179 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:54:03.187 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:54:03.193 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:54:03.194 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:54:03.214 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:54:03.217 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:54:03.219 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:54:03.251 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:54:03.257 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:54:03.258 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:54:03.281 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:54:03.287 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:54:03.289 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:54:03.312 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.319 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.321 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.323 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.328 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.333 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.339 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:54:03.341 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:54:03.348 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:54:03.348 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:54:03.367 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:54:03.377 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:54:03.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:54:03.408 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:54:03.418 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:54:03.418 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:54:03.436 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:54:03.446 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:54:03.449 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:54:03.465 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:03.545 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:03.579 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:03.670 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:03.703 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:54:03.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:54:03.770 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:03.823 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:03.876 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:03.923 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:03.964 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T11:54:03.964 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:54:04.033 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:04.089 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:04.142 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:04.198 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:04.229 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:54:04.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:54:04.297 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:04.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:54:04.661 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:04.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:54:04.750 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T11:54:04.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:54:04.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-09-13T11:54:04.821 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-09-13T11:54:04.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-13T11:54:04.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-09-13T11:54:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:54:04.880 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:54:04.881 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:54:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:54:05.365 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:54:05.375 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:54:05.375 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:54:05.469 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:54:05.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-09-13T11:54:05.547 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-09-13T11:54:05.549 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-09-13T11:54:05.601 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-09-13T11:54:05.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-09-13T11:54:05.656 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-09-13T11:54:05.658 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-09-13T11:54:05.735 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-09-13T11:54:05.737 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:54:05.737 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-09-13T11:54:05.765 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-09-13T11:54:05.765 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-09-13T11:54:05.798 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-09-13T11:54:05.798 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-09-13T11:54:05.827 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-09-13T11:54:05.827 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-09-13T11:54:05.850 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-09-13T11:54:05.850 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2024-09-13T11:54:05.850 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:54:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:54:05.881 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:54:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:54:05.911 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:54:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:54:05.941 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:54:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:54:05.970 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:54:05.970 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:54:06.007 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:54:06.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:54:06.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:54:06.110 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:54:06.120 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T11:54:06.123 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T11:54:06.276 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:54:06.287 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:54:06.288 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi037.front.sepia.ceph.com,smithi039.front.sepia.ceph.com,smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:58:00.290 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2024-09-13T11:58:00.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:58:00.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:58:00.380 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-09-13T11:58:00.462 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-09-13T11:58:00.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:58:00.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:58:00.548 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-09-13T11:58:00.632 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-09-13T11:58:00.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:58:00.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:58:00.718 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-13T11:58:00.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-13T11:58:00.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:58:00.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:58:00.880 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-09-13T11:58:00.966 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-09-13T11:58:00.967 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:58:00.977 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:58:00.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:58:00.978 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:58:00.981 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:58:00.981 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:58:00.983 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:58:00.983 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:58:00.986 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:58:00.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:58:01.013 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:58:01.017 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:58:01.018 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:58:01.030 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:58:01.033 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:58:01.034 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:58:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:58:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:58:01.057 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-09-13T11:58:01.061 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-09-13T11:58:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-09-13T11:58:01.070 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-09-13T11:58:01.074 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-09-13T11:58:01.075 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-09-13T11:58:01.080 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-09-13T11:58:01.085 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:58:01.089 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:58:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:58:01.091 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-09-13T11:58:01.101 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:58:01.104 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:58:01.105 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:58:01.106 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:58:01.121 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:58:01.160 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:58:01.162 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:58:01.163 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:58:01.164 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-09-13T11:58:01.164 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.164 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.164 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.164 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.165 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:58:01.165 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-13T11:58:01.165 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.166 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.166 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.166 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.169 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.173 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.183 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.187 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:58:01.188 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:58:01.196 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:58:01.197 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:58:01.197 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:58:01.197 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:58:01.202 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:58:01.202 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:58:01.203 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:58:01.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:58:01.205 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:58:01.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:58:01.206 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:58:01.206 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:58:01.207 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:58:01.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:58:01.362 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:58:01.363 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:58:01.374 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:58:01.374 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:58:01.386 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:58:01.386 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:58:01.387 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:58:01.387 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:58:01.500 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:58:01.500 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T11:58:01.500 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:58:01.504 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:58:01.504 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T11:58:01.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:58:01.512 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:58:01.512 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T11:58:01.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:58:01.514 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:58:01.515 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T11:58:01.515 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:58:01.527 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:58:01.527 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:58:01.533 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:58:01.533 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:58:01.543 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:58:01.543 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:58:01.544 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:58:01.545 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:58:01.595 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:58:01.600 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:58:01.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:58:01.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:58:01.673 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:58:01.677 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:58:01.687 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:58:01.689 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:58:01.739 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-09-13T11:58:01.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-09-13T11:58:01.743 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2024-09-13T11:58:01.745 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2024-09-13T11:58:01.752 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-09-13T11:58:01.754 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-09-13T11:58:01.754 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-09-13T11:58:01.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-09-13T11:58:01.977 INFO:teuthology.orchestra.run.smithi037.stdout:45 files removed 2024-09-13T11:58:01.987 INFO:teuthology.orchestra.run.smithi087.stdout:45 files removed 2024-09-13T11:58:01.988 INFO:teuthology.orchestra.run.smithi005.stdout:45 files removed 2024-09-13T11:58:01.991 INFO:teuthology.orchestra.run.smithi039.stdout:45 files removed 2024-09-13T11:58:02.003 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-09-13T11:58:02.016 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-09-13T11:58:02.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw 2024-09-13T11:58:02.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2024-09-13T11:58:02.569 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-13T11:58:02.581 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-13T11:58:02.586 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 316 kB/s | 84 kB 00:00 2024-09-13T11:58:02.589 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 312 kB/s | 84 kB 00:00 2024-09-13T11:58:02.853 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-13T11:58:02.878 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-13T11:58:02.881 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-13T11:58:02.882 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-13T11:58:03.121 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-13T11:58:03.145 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-13T11:58:03.148 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-13T11:58:03.158 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-13T11:58:04.007 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-13T11:58:04.883 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-13T11:58:05.695 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.2 MB 00:02 2024-09-13T11:58:05.842 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-13T11:58:06.070 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-13T11:58:06.734 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-13T11:58:07.488 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-13T11:58:11.219 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T11:58:11.260 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-13T11:58:11.931 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T11:58:12.802 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-13T11:58:12.802 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-13T11:58:13.415 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-13T11:58:13.600 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T11:58:13.907 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-13T11:58:14.219 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T11:58:15.155 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-13T11:58:19.297 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-09-13T11:58:19.513 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T11:58:19.941 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T11:58:21.464 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-13T11:58:21.770 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:58:21.792 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:58:21.793 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:58:21.794 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Install 30 Packages 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 2 Packages 2024-09-13T11:58:21.795 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 62 M 2024-09-13T11:58:21.796 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:58:21.959 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:58:21.968 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:21.969 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:58:21.970 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout:Install 30 Packages 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:21.971 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 62 M 2024-09-13T11:58:21.972 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:58:22.405 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:58:22.409 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:58:22.410 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:58:22.411 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout:Install 30 Packages 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2024-09-13T11:58:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:22.413 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 62 M 2024-09-13T11:58:22.413 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:58:22.581 INFO:teuthology.orchestra.run.smithi037.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-13T11:58:22.639 INFO:teuthology.orchestra.run.smithi037.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 433 kB/s | 25 kB 00:00 2024-09-13T11:58:22.739 INFO:teuthology.orchestra.run.smithi037.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-13T11:58:22.840 INFO:teuthology.orchestra.run.smithi037.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 5.0 MB/s | 508 kB 00:00 2024-09-13T11:58:23.131 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-13T11:58:23.132 INFO:teuthology.orchestra.run.smithi037.stdout:(5/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 10 MB/s | 11 MB 00:01 2024-09-13T11:58:23.357 INFO:teuthology.orchestra.run.smithi037.stdout:(6/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 17 MB/s | 22 MB 00:01 2024-09-13T11:58:23.390 INFO:teuthology.orchestra.run.smithi037.stdout:(7/32): python3-ceph-argparse-19.3.0-4697.gd58f 175 kB/s | 45 kB 00:00 2024-09-13T11:58:23.474 INFO:teuthology.orchestra.run.smithi037.stdout:(8/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 8.7 MB/s | 5.5 MB 00:00 2024-09-13T11:58:23.499 INFO:teuthology.orchestra.run.smithi037.stdout:(9/32): python3-ceph-common-19.3.0-4697.gd58f73 1.0 MB/s | 143 kB 00:00 2024-09-13T11:58:23.524 INFO:teuthology.orchestra.run.smithi037.stdout:(10/32): python3-cephfs-19.3.0-4697.gd58f7306.e 1.2 MB/s | 161 kB 00:00 2024-09-13T11:58:23.558 INFO:teuthology.orchestra.run.smithi037.stdout:(11/32): python3-rados-19.3.0-4697.gd58f7306.el 3.7 MB/s | 321 kB 00:00 2024-09-13T11:58:23.591 INFO:teuthology.orchestra.run.smithi037.stdout:(12/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 3.2 MB/s | 305 kB 00:00 2024-09-13T11:58:23.617 INFO:teuthology.orchestra.run.smithi037.stdout:(13/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.1 MB/s | 99 kB 00:00 2024-09-13T11:58:23.709 INFO:teuthology.orchestra.run.smithi037.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 711 kB/s | 107 kB 00:00 2024-09-13T11:58:23.734 INFO:teuthology.orchestra.run.smithi037.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-09-13T11:58:23.759 INFO:teuthology.orchestra.run.smithi037.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-09-13T11:58:23.773 INFO:teuthology.orchestra.run.smithi005.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 3.5 MB/s | 5.5 MB 00:01 2024-09-13T11:58:23.809 INFO:teuthology.orchestra.run.smithi037.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-13T11:58:23.824 INFO:teuthology.orchestra.run.smithi005.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 504 kB/s | 25 kB 00:00 2024-09-13T11:58:23.851 INFO:teuthology.orchestra.run.smithi037.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-13T11:58:23.876 INFO:teuthology.orchestra.run.smithi039.stdout:(1/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 17 MB/s | 22 MB 00:01 2024-09-13T11:58:23.876 INFO:teuthology.orchestra.run.smithi037.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-13T11:58:23.899 INFO:teuthology.orchestra.run.smithi005.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 9.9 MB/s | 755 kB 00:00 2024-09-13T11:58:23.902 INFO:teuthology.orchestra.run.smithi037.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-13T11:58:23.926 INFO:teuthology.orchestra.run.smithi039.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 507 kB/s | 25 kB 00:00 2024-09-13T11:58:23.927 INFO:teuthology.orchestra.run.smithi037.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-13T11:58:23.952 INFO:teuthology.orchestra.run.smithi037.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-09-13T11:58:23.974 INFO:teuthology.orchestra.run.smithi005.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 6.6 MB/s | 508 kB 00:00 2024-09-13T11:58:23.984 INFO:teuthology.orchestra.run.smithi039.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 13 MB/s | 755 kB 00:00 2024-09-13T11:58:24.035 INFO:teuthology.orchestra.run.smithi039.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 9.9 MB/s | 508 kB 00:00 2024-09-13T11:58:24.037 INFO:teuthology.orchestra.run.smithi037.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-13T11:58:24.061 INFO:teuthology.orchestra.run.smithi037.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 280 kB/s | 26 kB 00:00 2024-09-13T11:58:24.086 INFO:teuthology.orchestra.run.smithi037.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-09-13T11:58:24.112 INFO:teuthology.orchestra.run.smithi037.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T11:58:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T11:58:24.187 INFO:teuthology.orchestra.run.smithi037.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-13T11:58:24.235 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-13T11:58:24.260 INFO:teuthology.orchestra.run.smithi039.stdout:(5/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 24 MB/s | 5.5 MB 00:00 2024-09-13T11:58:24.263 INFO:teuthology.orchestra.run.smithi037.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-13T11:58:24.311 INFO:teuthology.orchestra.run.smithi039.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 905 kB/s | 45 kB 00:00 2024-09-13T11:58:24.355 INFO:teuthology.orchestra.run.smithi037.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-13T11:58:24.361 INFO:teuthology.orchestra.run.smithi039.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.8 MB/s | 143 kB 00:00 2024-09-13T11:58:24.391 INFO:teuthology.orchestra.run.smithi005.stdout:(5/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 13 MB/s | 5.5 MB 00:00 2024-09-13T11:58:24.436 INFO:teuthology.orchestra.run.smithi039.stdout:(8/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 3.0 MB/s | 5.5 MB 00:01 2024-09-13T11:58:24.458 INFO:teuthology.orchestra.run.smithi005.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 679 kB/s | 45 kB 00:00 2024-09-13T11:58:24.461 INFO:teuthology.orchestra.run.smithi039.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 1.6 MB/s | 161 kB 00:00 2024-09-13T11:58:24.486 INFO:teuthology.orchestra.run.smithi039.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.2 MB/s | 321 kB 00:00 2024-09-13T11:58:24.488 INFO:teuthology.orchestra.run.smithi037.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 14 MB/s | 3.2 MB 00:00 2024-09-13T11:58:24.508 INFO:teuthology.orchestra.run.smithi005.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.8 MB/s | 143 kB 00:00 2024-09-13T11:58:24.520 INFO:teuthology.orchestra.run.smithi039.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T11:58:24.545 INFO:teuthology.orchestra.run.smithi039.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T11:58:24.547 INFO:teuthology.orchestra.run.smithi037.stdout:(32/32): librados2-19.3.0-4697.gd58f7306.el9.x8 9.6 MB/s | 3.5 MB 00:00 2024-09-13T11:58:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-13T11:58:24.559 INFO:teuthology.orchestra.run.smithi005.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 3.2 MB/s | 161 kB 00:00 2024-09-13T11:58:24.609 INFO:teuthology.orchestra.run.smithi005.stdout:(9/32): python3-rados-19.3.0-4697.gd58f7306.el9 6.3 MB/s | 321 kB 00:00 2024-09-13T11:58:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:58:24.667 INFO:teuthology.orchestra.run.smithi005.stdout:(10/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T11:58:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:58:24.695 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:58:24.696 INFO:teuthology.orchestra.run.smithi039.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 610 kB/s | 107 kB 00:00 2024-09-13T11:58:24.718 INFO:teuthology.orchestra.run.smithi005.stdout:(11/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T11:58:24.737 INFO:teuthology.orchestra.run.smithi039.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 836 kB/s | 160 kB 00:00 2024-09-13T11:58:24.763 INFO:teuthology.orchestra.run.smithi039.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 675 kB/s | 45 kB 00:00 2024-09-13T11:58:24.863 INFO:teuthology.orchestra.run.smithi039.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-09-13T11:58:24.897 INFO:teuthology.orchestra.run.smithi039.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-13T11:58:24.918 INFO:teuthology.orchestra.run.smithi005.stdout:(12/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9 4.0 MB/s | 11 MB 00:02 2024-09-13T11:58:24.922 INFO:teuthology.orchestra.run.smithi039.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-09-13T11:58:24.943 INFO:teuthology.orchestra.run.smithi005.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 474 kB/s | 107 kB 00:00 2024-09-13T11:58:24.956 INFO:teuthology.orchestra.run.smithi039.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-13T11:58:24.981 INFO:teuthology.orchestra.run.smithi039.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-13T11:58:25.014 INFO:teuthology.orchestra.run.smithi039.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-09-13T11:58:25.019 INFO:teuthology.orchestra.run.smithi005.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-13T11:58:25.056 INFO:teuthology.orchestra.run.smithi039.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-13T11:58:25.078 INFO:teuthology.orchestra.run.smithi005.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-13T11:58:25.081 INFO:teuthology.orchestra.run.smithi039.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-13T11:58:25.107 INFO:teuthology.orchestra.run.smithi039.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-13T11:58:25.132 INFO:teuthology.orchestra.run.smithi039.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-13T11:58:25.175 INFO:teuthology.orchestra.run.smithi039.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-13T11:58:25.208 INFO:teuthology.orchestra.run.smithi039.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T11:58:25.245 INFO:teuthology.orchestra.run.smithi005.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-09-13T11:58:25.275 INFO:teuthology.orchestra.run.smithi039.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-13T11:58:25.303 INFO:teuthology.orchestra.run.smithi005.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 726 kB/s | 42 kB 00:00 2024-09-13T11:58:25.329 INFO:teuthology.orchestra.run.smithi005.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-09-13T11:58:25.358 INFO:teuthology.orchestra.run.smithi039.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-13T11:58:25.387 INFO:teuthology.orchestra.run.smithi005.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 618 kB/s | 36 kB 00:00 2024-09-13T11:58:25.400 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:58:25.401 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:58:25.412 INFO:teuthology.orchestra.run.smithi005.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-13T11:58:25.463 INFO:teuthology.orchestra.run.smithi005.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 625 kB/s | 47 kB 00:00 2024-09-13T11:58:25.497 INFO:teuthology.orchestra.run.smithi005.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.6 MB/s | 308 kB 00:00 2024-09-13T11:58:25.517 INFO:teuthology.orchestra.run.smithi039.stdout:(30/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 20 MB/s | 3.2 MB 00:00 2024-09-13T11:58:25.522 INFO:teuthology.orchestra.run.smithi005.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-13T11:58:25.547 INFO:teuthology.orchestra.run.smithi005.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-13T11:58:25.573 INFO:teuthology.orchestra.run.smithi005.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-13T11:58:25.623 INFO:teuthology.orchestra.run.smithi005.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-09-13T11:58:25.648 INFO:teuthology.orchestra.run.smithi005.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-13T11:58:25.684 INFO:teuthology.orchestra.run.smithi039.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 8.5 MB/s | 3.5 MB 00:00 2024-09-13T11:58:25.715 INFO:teuthology.orchestra.run.smithi005.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-09-13T11:58:25.791 INFO:teuthology.orchestra.run.smithi005.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-13T11:58:26.007 INFO:teuthology.orchestra.run.smithi005.stdout:(30/32): librados2-19.3.0-4697.gd58f7306.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-13T11:58:26.299 INFO:teuthology.orchestra.run.smithi005.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 6.4 MB/s | 3.2 MB 00:00 2024-09-13T11:58:26.801 INFO:teuthology.orchestra.run.smithi039.stdout:(32/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9 2.6 MB/s | 11 MB 00:04 2024-09-13T11:58:26.806 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:26.806 INFO:teuthology.orchestra.run.smithi039.stdout:Total 14 MB/s | 62 MB 00:04 2024-09-13T11:58:26.918 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:58:26.941 INFO:teuthology.orchestra.run.smithi005.stdout:(32/32): ceph-common-19.3.0-4697.gd58f7306.el9. 4.5 MB/s | 22 MB 00:04 2024-09-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:26.947 INFO:teuthology.orchestra.run.smithi005.stdout:Total 12 MB/s | 62 MB 00:05 2024-09-13T11:58:26.959 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:58:26.959 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:58:27.060 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:58:27.101 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:58:27.101 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:58:27.199 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:58:27.287 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:58:27.319 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:58:27.352 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:58:27.375 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:27.398 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:27.544 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:27.601 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:27.667 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:58:27.667 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:58:27.841 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:58:27.841 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:58:27.965 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:58:28.070 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:28.146 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:28.236 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:58:28.263 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:28.326 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:28.355 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:58:28.399 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:58:28.469 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:28.563 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:28.597 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:58:28.710 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:58:28.740 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:58:28.781 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:58:28.821 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:58:28.851 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:58:28.883 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:58:29.191 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:58:29.246 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:58:29.260 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:58:29.278 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:29.289 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:58:29.322 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:58:29.345 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:29.356 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:58:29.366 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:29.380 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:29.409 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:58:29.451 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:58:29.453 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:58:29.480 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:58:29.505 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:58:29.528 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:29.557 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:29.572 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:29.627 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:29.658 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T11:58:29.757 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:29.810 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:29.973 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:58:30.006 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:58:30.078 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:30.107 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:30.159 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:58:30.192 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:58:30.201 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:30.219 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:58:30.273 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:30.290 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:58:30.317 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:30.380 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:30.383 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:30.417 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:58:30.453 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:58:30.474 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:58:30.499 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:30.526 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:30.552 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:30.562 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:30.591 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:58:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:30.627 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:58:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:58:30.708 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:30.735 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:30.746 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:58:30.785 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:58:30.800 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:30.834 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:58:30.835 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:58:30.867 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:58:30.896 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:58:30.946 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:58:30.946 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:58:30.985 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:58:31.033 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:58:31.067 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:58:31.096 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:58:31.129 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:58:31.283 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:58:31.333 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:31.426 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:31.458 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:58:31.467 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:58:31.500 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:58:31.516 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:31.600 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:31.632 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:58:31.675 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:58:32.019 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:58:32.051 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:58:32.054 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:58:32.055 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:58:32.056 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:58:32.057 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Install 30 Packages 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-09-13T11:58:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:32.059 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 62 M 2024-09-13T11:58:32.059 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:58:32.060 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:32.116 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:32.194 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:58:32.197 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:58:32.218 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:32.230 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:58:32.291 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:32.372 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:58:32.413 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:58:32.664 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:32.696 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:32.943 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 7.8 MB/s | 5.5 MB 00:00 2024-09-13T11:58:32.993 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 503 kB/s | 25 kB 00:00 2024-09-13T11:58:33.077 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.9 MB/s | 755 kB 00:00 2024-09-13T11:58:33.160 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 6.0 MB/s | 508 kB 00:00 2024-09-13T11:58:33.299 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:33.335 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 10 MB/s | 11 MB 00:01 2024-09-13T11:58:33.341 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:33.377 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 1.1 MB/s | 45 kB 00:00 2024-09-13T11:58:33.433 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:33.433 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:58:33.433 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:58:33.433 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:33.444 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.1 MB/s | 143 kB 00:00 2024-09-13T11:58:33.467 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:33.511 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.4 MB/s | 161 kB 00:00 2024-09-13T11:58:33.611 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 12 MB/s | 5.5 MB 00:00 2024-09-13T11:58:33.636 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 2.5 MB/s | 321 kB 00:00 2024-09-13T11:58:33.662 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 6.0 MB/s | 305 kB 00:00 2024-09-13T11:58:33.687 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T11:58:33.896 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 13 MB/s | 22 MB 00:01 2024-09-13T11:58:34.288 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 171 kB/s | 107 kB 00:00 2024-09-13T11:58:34.313 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 108 kB/s | 45 kB 00:00 2024-09-13T11:58:34.372 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 234 kB/s | 160 kB 00:00 2024-09-13T11:58:34.497 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 338 kB/s | 42 kB 00:00 2024-09-13T11:58:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:34.747 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 674 kB/s | 292 kB 00:00 2024-09-13T11:58:34.822 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:34.868 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:34.972 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 968 kB/s | 662 kB 00:00 2024-09-13T11:58:34.997 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 441 kB/s | 221 kB 00:00 2024-09-13T11:58:35.023 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 130 kB/s | 36 kB 00:00 2024-09-13T11:58:35.081 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.6 MB/s | 308 kB 00:00 2024-09-13T11:58:35.107 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 354 kB/s | 47 kB 00:00 2024-09-13T11:58:35.132 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 538 kB/s | 26 kB 00:00 2024-09-13T11:58:35.165 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-13T11:58:35.199 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-09-13T11:58:35.233 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T11:58:35.266 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.3 MB/s | 839 kB 00:00 2024-09-13T11:58:35.350 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:35.358 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-13T11:58:35.389 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:35.400 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-13T11:58:35.463 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:35.463 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:58:35.463 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:58:35.464 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:35.486 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:35.497 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:35.529 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:35.592 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-13T11:58:35.620 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:35.621 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:58:35.621 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:58:35.621 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:35.650 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-13T11:58:35.656 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:35.656 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:35.656 INFO:teuthology.orchestra.run.smithi087.stdout:Total 17 MB/s | 62 MB 00:03 2024-09-13T11:58:35.761 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:58:35.796 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:58:35.796 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:58:36.488 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:58:36.489 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:58:37.966 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:58:38.086 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:58:38.139 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:58:38.164 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:58:38.187 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:38.211 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:58:38.406 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:38.460 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:58:38.803 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:58:38.915 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:39.024 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:39.121 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:58:39.159 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:39.222 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:58:39.259 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:58:39.302 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:58:39.388 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:39.411 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:39.475 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:58:39.509 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:58:39.617 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:58:39.652 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:58:39.693 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:58:39.736 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:58:39.776 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:58:39.815 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:58:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:58:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:40.300 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:58:40.330 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:58:40.375 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:58:40.896 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:58:40.936 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:41.007 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:58:41.086 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:58:41.113 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:58:43.379 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:43.418 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:43.822 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-09-13T11:58:43.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:44.012 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:58:44.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:44.153 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:58:44.153 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:58:44.153 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:58:44.153 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:44.197 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:44.865 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:44.892 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:44.892 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:58:44.892 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:58:44.893 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:44.893 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:44.893 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:44.894 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:44.985 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:44.985 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /root 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2024-09-13T11:58:45.970 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-09-13T11:58:46.171 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:58:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:58:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:58:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:58:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:58:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:58:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:58:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:58:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:58:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:58:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:58:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:58:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:58:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:58:47.000 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:47.027 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:47.027 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:58:47.027 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:58:47.027 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:47.028 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:47.028 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:47.029 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:47.118 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:47.223 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:47.250 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:58:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:58:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:47.250 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:47.251 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:47.252 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:47.326 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:47.524 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:58:47.525 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:58:47.526 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:58:47.527 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:47.528 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:58:47.858 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-09-13T11:58:48.315 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:48.315 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:58:48.315 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:58:48.316 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:58:48.317 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:58:48.318 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:58:48.318 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:58:48.318 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:58:48.319 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:58:48.319 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:48.319 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:58:48.319 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:58:48.462 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:58:48.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:58:48.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:58:48.545 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:58:48.545 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:48.545 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:58:48.545 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:58:48.601 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:58:48.601 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:48.601 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:58:48.601 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-09-13T11:58:48.602 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:48.603 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 51 M 2024-09-13T11:58:48.603 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 216 M 2024-09-13T11:58:48.603 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:58:48.872 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-13T11:58:48.906 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-13T11:58:48.956 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 727 kB/s | 161 kB 00:00 2024-09-13T11:58:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.403 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:58:49.404 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:58:49.405 INFO:teuthology.orchestra.run.smithi005.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:58:49.406 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:49.406 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.483 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:58:49.484 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:58:49.485 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:58:49.486 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:58:49.487 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:58:49.487 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:49.487 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:58:49.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-test 2024-09-13T11:58:49.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2024-09-13T11:58:50.331 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-13T11:58:50.333 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:50.334 INFO:teuthology.orchestra.run.smithi037.stdout:Total 29 MB/s | 51 MB 00:01 2024-09-13T11:58:50.334 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:31 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:58:50.363 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:58:50.376 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:31 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:58:50.385 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:58:50.385 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:58:50.482 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:50.483 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:58:50.484 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 51 M 2024-09-13T11:58:50.485 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 216 M 2024-09-13T11:58:50.485 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:58:50.514 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:58:50.515 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2024-09-13T11:58:50.516 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:58:50.517 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 51 M 2024-09-13T11:58:50.517 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 216 M 2024-09-13T11:58:50.517 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:58:50.724 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 744 kB/s | 161 kB 00:00 2024-09-13T11:58:50.786 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:58:50.786 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:58:50.872 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T11:58:50.897 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 668 kB/s | 161 kB 00:00 2024-09-13T11:58:50.922 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T11:58:51.300 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 384 kB/s | 304 kB 00:00 2024-09-13T11:58:51.333 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 105 kB/s | 64 kB 00:00 2024-09-13T11:58:51.344 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:58:51.430 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:58:51.475 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:58:51.499 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:52.125 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 31 MB/s | 50 MB 00:01 2024-09-13T11:58:52.127 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:52.128 INFO:teuthology.orchestra.run.smithi005.stdout:Total 31 MB/s | 51 MB 00:01 2024-09-13T11:58:52.156 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:58:52.179 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:58:52.179 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:58:52.580 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:58:52.580 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:58:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:58:53.220 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:58:53.263 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:58:53.289 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 19 MB/s | 50 MB 00:02 2024-09-13T11:58:53.291 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:58:53.291 INFO:teuthology.orchestra.run.smithi039.stdout:Total 18 MB/s | 51 MB 00:02 2024-09-13T11:58:53.293 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:53.320 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:58:53.343 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:58:53.343 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:58:53.744 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:58:53.745 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-09-13T11:58:54.172 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:54.297 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:58:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:58:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:58:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:55.179 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:55.204 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:58:55.204 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:58:55.204 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:58:55.205 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:55.205 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:58:55.205 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:55.206 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:55.272 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:58:55.273 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:58:56.435 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:58:56.436 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:58:56.437 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:58:56.438 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:58:56.438 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:58:56.438 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:58:56.438 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:58:56.663 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:58:57.412 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:58:57.412 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:57.412 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-09-13T11:58:57.412 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.412 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:58:57.413 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.414 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:57.415 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:58:57.777 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-09-13T11:58:58.326 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:58.338 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:58:58.403 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:58:58.472 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:58:58.472 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2024-09-13T11:58:58.473 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:58:58.474 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 51 M 2024-09-13T11:58:58.474 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 216 M 2024-09-13T11:58:58.474 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:58:58.719 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 722 kB/s | 161 kB 00:00 2024-09-13T11:58:58.744 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-13T11:58:58.836 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 546 kB/s | 64 kB 00:00 2024-09-13T11:58:59.212 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:58:59.212 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:58:59.212 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:58:59.212 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:58:59.483 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:58:59.550 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:58:59.558 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:58:59.818 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-09-13T11:59:00.040 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T11:59:00.042 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:00.043 INFO:teuthology.orchestra.run.smithi087.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-13T11:59:00.071 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:00.091 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:00.091 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:00.410 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:00.473 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:00.473 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:00.561 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:59:00.563 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:59:00.564 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:59:00.565 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:59:00.566 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:59:00.567 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:00.567 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:00.567 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:00.567 INFO:teuthology.orchestra.run.smithi037.stdout:Install 37 Packages 2024-09-13T11:59:00.567 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:00.568 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 M 2024-09-13T11:59:00.568 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 109 M 2024-09-13T11:59:00.568 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:00.800 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:59:00.800 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:59:00.801 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:59:00.801 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:59:00.998 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:01.070 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:59:01.097 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:59:01.097 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:01.097 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:01.098 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:01.125 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:59:01.154 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:59:01.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph 2024-09-13T11:59:01.423 INFO:teuthology.orchestra.run.smithi037.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-13T11:59:01.631 INFO:teuthology.orchestra.run.smithi037.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 4.4 MB/s | 1.5 MB 00:00 2024-09-13T11:59:01.690 INFO:teuthology.orchestra.run.smithi037.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 5.8 MB/s | 2.4 MB 00:00 2024-09-13T11:59:01.756 INFO:teuthology.orchestra.run.smithi037.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 4.3 MB/s | 291 kB 00:00 2024-09-13T11:59:01.923 INFO:teuthology.orchestra.run.smithi037.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-13T11:59:01.949 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:59:01.949 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:59:01.949 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:59:01.949 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:59:01.964 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:42 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:02.107 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:02.109 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:59:02.110 INFO:teuthology.orchestra.run.smithi005.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:59:02.111 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:59:02.112 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout:Install 37 Packages 2024-09-13T11:59:02.113 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:02.114 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 31 M 2024-09-13T11:59:02.114 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 109 M 2024-09-13T11:59:02.114 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:02.174 INFO:teuthology.orchestra.run.smithi037.stdout:(6/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 32 MB/s | 17 MB 00:00 2024-09-13T11:59:02.215 INFO:teuthology.orchestra.run.smithi037.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 4.2 MB/s | 1.2 MB 00:00 2024-09-13T11:59:02.227 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:59:02.227 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:02.227 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:02.228 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:02.266 INFO:teuthology.orchestra.run.smithi037.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-13T11:59:02.333 INFO:teuthology.orchestra.run.smithi037.stdout:(9/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 5.3 MB/s | 4.8 MB 00:00 2024-09-13T11:59:02.358 INFO:teuthology.orchestra.run.smithi037.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-09-13T11:59:02.383 INFO:teuthology.orchestra.run.smithi037.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-13T11:59:02.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2024-09-13T11:59:02.500 INFO:teuthology.orchestra.run.smithi037.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-13T11:59:02.535 INFO:teuthology.orchestra.run.smithi005.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-13T11:59:02.600 INFO:teuthology.orchestra.run.smithi037.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 794 kB/s | 172 kB 00:00 2024-09-13T11:59:02.625 INFO:teuthology.orchestra.run.smithi037.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 277 kB/s | 35 kB 00:00 2024-09-13T11:59:02.651 INFO:teuthology.orchestra.run.smithi037.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 644 kB/s | 188 kB 00:00 2024-09-13T11:59:02.676 INFO:teuthology.orchestra.run.smithi037.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-13T11:59:02.701 INFO:teuthology.orchestra.run.smithi037.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-13T11:59:02.702 INFO:teuthology.orchestra.run.smithi005.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-13T11:59:02.752 INFO:teuthology.orchestra.run.smithi037.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-13T11:59:02.777 INFO:teuthology.orchestra.run.smithi037.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-13T11:59:02.778 INFO:teuthology.orchestra.run.smithi005.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-13T11:59:02.802 INFO:teuthology.orchestra.run.smithi037.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 865 kB/s | 43 kB 00:00 2024-09-13T11:59:02.853 INFO:teuthology.orchestra.run.smithi037.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 8.3 MB/s | 358 kB 00:00 2024-09-13T11:59:02.878 INFO:teuthology.orchestra.run.smithi037.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 470 kB/s | 11 kB 00:00 2024-09-13T11:59:02.895 INFO:teuthology.orchestra.run.smithi037.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-13T11:59:02.903 INFO:teuthology.orchestra.run.smithi005.stdout:(4/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-13T11:59:02.920 INFO:teuthology.orchestra.run.smithi037.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T11:59:02.928 INFO:teuthology.orchestra.run.smithi005.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 1.9 MB/s | 291 kB 00:00 2024-09-13T11:59:02.937 INFO:teuthology.orchestra.run.smithi037.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 552 kB/s | 23 kB 00:00 2024-09-13T11:59:02.962 INFO:teuthology.orchestra.run.smithi037.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2024-09-13T11:59:02.979 INFO:teuthology.orchestra.run.smithi037.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 573 kB/s | 20 kB 00:00 2024-09-13T11:59:02.996 INFO:teuthology.orchestra.run.smithi037.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-13T11:59:03.022 INFO:teuthology.orchestra.run.smithi037.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-13T11:59:03.047 INFO:teuthology.orchestra.run.smithi037.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 680 kB/s | 16 kB 00:00 2024-09-13T11:59:03.072 INFO:teuthology.orchestra.run.smithi037.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-13T11:59:03.074 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:44 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:03.079 INFO:teuthology.orchestra.run.smithi005.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 8.2 MB/s | 1.2 MB 00:00 2024-09-13T11:59:03.097 INFO:teuthology.orchestra.run.smithi037.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T11:59:03.103 INFO:teuthology.orchestra.run.smithi005.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-13T11:59:03.122 INFO:teuthology.orchestra.run.smithi037.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-13T11:59:03.137 INFO:teuthology.orchestra.run.smithi005.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-13T11:59:03.151 INFO:teuthology.orchestra.run.smithi037.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.2 MB/s | 230 kB 00:00 2024-09-13T11:59:03.176 INFO:teuthology.orchestra.run.smithi037.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-13T11:59:03.201 INFO:teuthology.orchestra.run.smithi037.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.3 MB/s | 427 kB 00:00 2024-09-13T11:59:03.222 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:03.225 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:03.225 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:03.225 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:03.225 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:03.225 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:59:03.226 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:59:03.227 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:59:03.227 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:59:03.227 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:59:03.227 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:59:03.227 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:59:03.228 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:59:03.229 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:59:03.230 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:59:03.230 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:59:03.230 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:59:03.230 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:59:03.230 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout:Install 37 Packages 2024-09-13T11:59:03.231 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:03.232 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 31 M 2024-09-13T11:59:03.232 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 109 M 2024-09-13T11:59:03.232 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:03.268 INFO:teuthology.orchestra.run.smithi037.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-09-13T11:59:03.273 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:03.273 INFO:teuthology.orchestra.run.smithi037.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-13T11:59:03.321 INFO:teuthology.orchestra.run.smithi005.stdout:(9/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-13T11:59:03.346 INFO:teuthology.orchestra.run.smithi005.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-09-13T11:59:03.371 INFO:teuthology.orchestra.run.smithi005.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 505 kB/s | 135 kB 00:00 2024-09-13T11:59:03.433 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:03.477 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:03.477 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:03.488 INFO:teuthology.orchestra.run.smithi005.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-13T11:59:03.617 INFO:teuthology.orchestra.run.smithi039.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-13T11:59:03.798 INFO:teuthology.orchestra.run.smithi039.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-13T11:59:03.813 INFO:teuthology.orchestra.run.smithi005.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 107 kB/s | 35 kB 00:00 2024-09-13T11:59:03.857 INFO:teuthology.orchestra.run.smithi039.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-13T11:59:03.949 INFO:teuthology.orchestra.run.smithi039.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 3.1 MB/s | 291 kB 00:00 2024-09-13T11:59:03.974 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:03.975 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:03.989 INFO:teuthology.orchestra.run.smithi005.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 293 kB/s | 188 kB 00:00 2024-09-13T11:59:04.014 INFO:teuthology.orchestra.run.smithi005.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 268 kB/s | 172 kB 00:00 2024-09-13T11:59:04.016 INFO:teuthology.orchestra.run.smithi039.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-13T11:59:04.101 INFO:teuthology.orchestra.run.smithi005.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 269 kB/s | 77 kB 00:00 2024-09-13T11:59:04.126 INFO:teuthology.orchestra.run.smithi005.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 372 kB/s | 51 kB 00:00 2024-09-13T11:59:04.152 INFO:teuthology.orchestra.run.smithi005.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 302 kB/s | 42 kB 00:00 2024-09-13T11:59:04.202 INFO:teuthology.orchestra.run.smithi005.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T11:59:04.228 INFO:teuthology.orchestra.run.smithi005.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 176 kB/s | 22 kB 00:00 2024-09-13T11:59:04.261 INFO:teuthology.orchestra.run.smithi005.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 400 kB/s | 43 kB 00:00 2024-09-13T11:59:04.286 INFO:teuthology.orchestra.run.smithi005.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 465 kB/s | 11 kB 00:00 2024-09-13T11:59:04.311 INFO:teuthology.orchestra.run.smithi005.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-09-13T11:59:04.337 INFO:teuthology.orchestra.run.smithi005.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T11:59:04.362 INFO:teuthology.orchestra.run.smithi005.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-13T11:59:04.366 INFO:teuthology.orchestra.run.smithi039.stdout:(6/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-13T11:59:04.387 INFO:teuthology.orchestra.run.smithi005.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-13T11:59:04.413 INFO:teuthology.orchestra.run.smithi005.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-13T11:59:04.438 INFO:teuthology.orchestra.run.smithi005.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-13T11:59:04.463 INFO:teuthology.orchestra.run.smithi005.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-13T11:59:04.489 INFO:teuthology.orchestra.run.smithi005.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-13T11:59:04.514 INFO:teuthology.orchestra.run.smithi005.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T11:59:04.539 INFO:teuthology.orchestra.run.smithi005.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-13T11:59:04.564 INFO:teuthology.orchestra.run.smithi005.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T11:59:04.590 INFO:teuthology.orchestra.run.smithi005.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T11:59:04.615 INFO:teuthology.orchestra.run.smithi005.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-13T11:59:04.640 INFO:teuthology.orchestra.run.smithi005.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T11:59:04.699 INFO:teuthology.orchestra.run.smithi005.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-09-13T11:59:04.702 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:04.703 INFO:teuthology.orchestra.run.smithi005.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-13T11:59:04.833 INFO:teuthology.orchestra.run.smithi039.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 286 kB/s | 253 kB 00:00 2024-09-13T11:59:04.860 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:04.904 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:04.905 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:05.100 INFO:teuthology.orchestra.run.smithi039.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 147 kB/s | 106 kB 00:00 2024-09-13T11:59:06.051 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:59:06.051 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:59:06.051 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:06.051 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:06.053 INFO:teuthology.orchestra.run.smithi039.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-13T11:59:06.053 INFO:teuthology.orchestra.run.smithi039.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-13T11:59:06.053 INFO:teuthology.orchestra.run.smithi039.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 946 kB/s | 188 kB 00:00 2024-09-13T11:59:06.053 INFO:teuthology.orchestra.run.smithi039.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 178 kB/s | 125 kB 00:00 2024-09-13T11:59:06.053 INFO:teuthology.orchestra.run.smithi039.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi039.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 694 kB/s | 35 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi039.stdout:(15/37): python3-urllib3-1.26.5-6.el9.noarch.rp 280 kB/s | 215 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi039.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi039.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 461 kB/s | 77 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi039.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:06.054 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:59:06.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:59:06.056 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:59:06.069 INFO:teuthology.orchestra.run.smithi039.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 248 kB/s | 42 kB 00:00 2024-09-13T11:59:06.095 INFO:teuthology.orchestra.run.smithi039.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 736 kB/s | 43 kB 00:00 2024-09-13T11:59:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:59:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:59:06.137 INFO:teuthology.orchestra.run.smithi039.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-13T11:59:06.157 INFO:teuthology.orchestra.run.smithi039.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 595 kB/s | 11 kB 00:00 2024-09-13T11:59:06.174 INFO:teuthology.orchestra.run.smithi039.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-13T11:59:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:59:06.207 INFO:teuthology.orchestra.run.smithi039.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T11:59:06.224 INFO:teuthology.orchestra.run.smithi039.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 549 kB/s | 23 kB 00:00 2024-09-13T11:59:06.239 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:59:06.241 INFO:teuthology.orchestra.run.smithi039.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-09-13T11:59:06.258 INFO:teuthology.orchestra.run.smithi039.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-13T11:59:06.277 INFO:teuthology.orchestra.run.smithi039.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-13T11:59:06.300 INFO:teuthology.orchestra.run.smithi039.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-09-13T11:59:06.315 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:59:06.317 INFO:teuthology.orchestra.run.smithi039.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 990 kB/s | 16 kB 00:00 2024-09-13T11:59:06.342 INFO:teuthology.orchestra.run.smithi039.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-13T11:59:06.368 INFO:teuthology.orchestra.run.smithi039.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T11:59:06.393 INFO:teuthology.orchestra.run.smithi039.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-13T11:59:06.419 INFO:teuthology.orchestra.run.smithi039.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-13T11:59:06.435 INFO:teuthology.orchestra.run.smithi039.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-13T11:59:06.457 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:59:06.460 INFO:teuthology.orchestra.run.smithi039.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-09-13T11:59:06.502 INFO:teuthology.orchestra.run.smithi039.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-09-13T11:59:06.507 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:06.508 INFO:teuthology.orchestra.run.smithi039.stdout:Total 9.4 MB/s | 31 MB 00:03 2024-09-13T11:59:06.541 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:59:06.592 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:59:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:59:06.668 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:59:06.712 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:06.712 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:06.761 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:59:06.827 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:06.902 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:59:06.935 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:59:06.962 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:59:06.963 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:59:07.008 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:59:07.040 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:59:07.088 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:59:07.088 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:59:07.120 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:59:07.178 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:59:07.198 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:07.199 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:07.216 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:59:07.218 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:59:07.303 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:59:07.349 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:59:07.357 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:59:07.392 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:59:07.421 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:59:07.464 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:59:07.497 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:59:07.528 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:59:07.539 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:59:07.559 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:07.582 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:59:07.585 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:07.586 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:07.586 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:59:07.586 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:07.586 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:07.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:07.616 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:59:07.666 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:59:07.714 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:59:07.800 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:59:07.955 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:59:08.037 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:59:08.096 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:59:08.133 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:59:08.192 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:59:08.256 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:59:08.463 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:59:08.512 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:59:08.524 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:59:08.524 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:59:08.524 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:59:08.524 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:59:08.594 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:59:08.692 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:59:08.733 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:08.737 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:59:08.797 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:59:08.831 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:59:08.858 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:59:08.875 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:59:08.895 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:59:08.895 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:08.895 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:08.896 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:08.934 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:59:08.982 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:59:09.015 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:59:09.049 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:59:09.079 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:59:09.105 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:09.106 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:09.111 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:59:09.160 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:09.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-09-13T11:59:09.188 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:09.188 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:09.189 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:59:09.189 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:09.189 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:09.189 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:09.313 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:59:09.360 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:59:09.394 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:59:09.424 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:59:09.458 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:59:09.492 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:59:09.533 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:59:09.576 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:59:09.610 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:59:09.659 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:59:09.660 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:09.686 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:09.686 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:09.686 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:59:09.687 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:09.687 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:09.687 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:09.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:59:09.746 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:59:09.890 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:59:09.893 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:59:09.894 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:59:09.895 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout:Install 37 Packages 2024-09-13T11:59:09.896 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:09.897 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 M 2024-09-13T11:59:09.897 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 109 M 2024-09-13T11:59:09.897 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:09.919 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:09.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:59:09.944 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:09.945 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:09.945 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:59:09.945 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:09.945 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:09.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:10.010 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:10.036 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:59:10.088 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:59:10.125 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:59:10.183 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:59:10.215 INFO:teuthology.orchestra.run.smithi087.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-13T11:59:10.248 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:59:10.390 INFO:teuthology.orchestra.run.smithi087.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-13T11:59:10.444 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:59:10.473 INFO:teuthology.orchestra.run.smithi087.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-13T11:59:10.495 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:59:10.557 INFO:teuthology.orchestra.run.smithi087.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 3.4 MB/s | 291 kB 00:00 2024-09-13T11:59:10.576 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:59:10.666 INFO:teuthology.orchestra.run.smithi087.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-13T11:59:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:59:10.708 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:10.711 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:59:10.738 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:10.739 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:10.739 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:59:10.739 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:10.739 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:10.739 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:10.777 INFO:teuthology.orchestra.run.smithi087.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-13T11:59:10.816 INFO:teuthology.orchestra.run.smithi087.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T11:59:10.848 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:59:10.984 INFO:teuthology.orchestra.run.smithi087.stdout:(8/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-13T11:59:11.017 INFO:teuthology.orchestra.run.smithi087.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-13T11:59:11.024 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:59:11.054 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:11.059 INFO:teuthology.orchestra.run.smithi087.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2024-09-13T11:59:11.081 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:11.081 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:11.082 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:59:11.082 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:11.082 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:11.082 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:11.084 INFO:teuthology.orchestra.run.smithi087.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T11:59:11.126 INFO:teuthology.orchestra.run.smithi087.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 876 kB/s | 125 kB 00:00 2024-09-13T11:59:11.209 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:11.226 INFO:teuthology.orchestra.run.smithi087.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-09-13T11:59:11.234 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:11.235 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:11.235 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:59:11.235 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:11.235 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:11.235 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:11.260 INFO:teuthology.orchestra.run.smithi087.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 981 kB/s | 172 kB 00:00 2024-09-13T11:59:11.285 INFO:teuthology.orchestra.run.smithi087.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 833 kB/s | 188 kB 00:00 2024-09-13T11:59:11.310 INFO:teuthology.orchestra.run.smithi087.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-13T11:59:11.344 INFO:teuthology.orchestra.run.smithi087.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 657 kB/s | 77 kB 00:00 2024-09-13T11:59:11.370 INFO:teuthology.orchestra.run.smithi087.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 496 kB/s | 42 kB 00:00 2024-09-13T11:59:11.420 INFO:teuthology.orchestra.run.smithi087.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T11:59:11.445 INFO:teuthology.orchestra.run.smithi087.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-13T11:59:11.470 INFO:teuthology.orchestra.run.smithi087.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.5 MB/s | 172 kB 00:00 2024-09-13T11:59:11.476 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:11.496 INFO:teuthology.orchestra.run.smithi087.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-09-13T11:59:11.501 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:11.501 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:11.502 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:59:11.502 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:11.502 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:11.502 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:11.521 INFO:teuthology.orchestra.run.smithi087.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-13T11:59:11.554 INFO:teuthology.orchestra.run.smithi087.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 910 kB/s | 23 kB 00:00 2024-09-13T11:59:11.567 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:11.580 INFO:teuthology.orchestra.run.smithi087.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 836 kB/s | 19 kB 00:00 2024-09-13T11:59:11.605 INFO:teuthology.orchestra.run.smithi087.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-13T11:59:11.630 INFO:teuthology.orchestra.run.smithi087.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-13T11:59:11.656 INFO:teuthology.orchestra.run.smithi087.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-13T11:59:11.689 INFO:teuthology.orchestra.run.smithi087.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-13T11:59:11.714 INFO:teuthology.orchestra.run.smithi087.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-09-13T11:59:11.740 INFO:teuthology.orchestra.run.smithi087.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-13T11:59:11.765 INFO:teuthology.orchestra.run.smithi087.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:59:11.778 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:59:11.779 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:59:11.780 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:59:11.781 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:59:11.790 INFO:teuthology.orchestra.run.smithi087.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-13T11:59:11.815 INFO:teuthology.orchestra.run.smithi087.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T11:59:11.841 INFO:teuthology.orchestra.run.smithi087.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-13T11:59:11.866 INFO:teuthology.orchestra.run.smithi087.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T11:59:11.891 INFO:teuthology.orchestra.run.smithi087.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T11:59:11.895 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:11.895 INFO:teuthology.orchestra.run.smithi087.stdout:Total 15 MB/s | 31 MB 00:01 2024-09-13T11:59:12.046 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:12.086 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:12.086 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:12.553 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:12.553 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:12.656 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:12.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:12.799 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:59:12.800 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:59:12.801 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:12.802 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:13.155 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-09-13T11:59:13.156 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:13.182 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:13.182 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:13.182 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:59:13.183 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:13.183 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:13.183 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:13.415 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:13.441 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:13.516 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:59:13.522 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:59:13.523 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:59:13.524 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:59:13.525 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:59:13.526 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:59:13.526 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:59:13.526 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:59:13.760 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:13.856 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:13.900 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:13.900 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:13.901 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T11:59:13.901 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:13.971 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-09-13T11:59:14.003 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:59:14.035 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:59:14.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:59:14.133 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:59:14.178 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:59:14.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:59:14.301 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:59:14.374 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:59:14.414 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:59:14.449 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:59:14.494 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:59:14.529 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:59:14.560 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:14.562 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:14.598 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:59:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:59:14.600 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:14.601 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:14.602 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:59:14.638 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:59:14.672 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:59:14.698 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:14.699 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 783 k 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 789 k 2024-09-13T11:59:14.700 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:59:14.753 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:59:14.843 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:59:14.943 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-base 2024-09-13T11:59:14.969 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.8 MB/s | 783 kB 00:00 2024-09-13T11:59:14.970 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:14.970 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-13T11:59:14.970 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:14.976 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:14.976 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:14.981 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:59:14.982 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:14.983 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:15.060 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:15.070 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:59:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:59:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:59:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:59:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:59:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:59:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:59:15.219 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:59:15.282 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:59:15.477 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:59:15.490 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:15.524 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:59:15.555 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:15.583 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:15.610 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:59:15.647 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:15.692 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:15.693 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T11:59:15.694 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:59:15.740 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:59:15.760 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install cephadm 2024-09-13T11:59:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:59:15.935 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:16.045 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:59:16.075 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:16.101 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:59:16.101 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:16.102 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:59:16.102 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:16.102 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:59:16.102 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:16.128 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:59:16.129 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:59:16.130 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:59:16.131 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:59:16.132 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:59:16.133 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:59:16.133 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:16.133 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:16.206 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:16.206 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:16.206 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:16.207 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:16.207 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:16.207 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:16.346 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:16.374 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:59:16.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2024-09-13T11:59:16.483 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:59:16.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 783 k 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 789 k 2024-09-13T11:59:16.485 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:16.751 INFO:teuthology.orchestra.run.smithi005.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:59:16.752 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:16.752 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T11:59:16.752 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:16.758 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:16.758 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:16.764 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:16.764 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:16.841 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:16.966 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:17.105 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:17.106 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:17.106 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-13T11:59:17.106 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:17.106 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 145 k 2024-09-13T11:59:17.107 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 432 k 2024-09-13T11:59:17.108 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:17.138 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:17.182 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:17.183 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T11:59:17.184 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:17.223 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:17.249 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2024-09-13T11:59:17.324 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 669 kB/s | 145 kB 00:00 2024-09-13T11:59:17.325 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:17.325 INFO:teuthology.orchestra.run.smithi037.stdout:Total 664 kB/s | 145 kB 00:00 2024-09-13T11:59:17.325 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:17.326 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:17.333 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:17.334 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:17.393 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:17.394 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:17.539 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:17.576 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:17.603 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:59:17.603 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:17.603 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:59:17.603 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:17.604 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:59:17.604 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:17.608 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:17.636 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:59:17.636 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:17.667 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:17.839 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:17.923 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:17.923 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:17.923 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:17.923 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:17.923 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:17.924 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:17.977 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:59:17.978 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 783 k 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 789 k 2024-09-13T11:59:17.979 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:18.052 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:18.075 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:59:18.076 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:18.076 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:59:18.076 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:18.076 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:59:18.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:18.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:59:18.232 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-13T11:59:18.233 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:18.233 INFO:teuthology.orchestra.run.smithi039.stdout:Total 3.0 MB/s | 783 kB 00:00 2024-09-13T11:59:18.233 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:18.239 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:18.239 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:18.245 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:18.245 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:18.294 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:59:18.318 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:18.320 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:18.598 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:18.708 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:18.796 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:18.830 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-09-13T11:59:18.845 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:59:18.846 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 145 k 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 432 k 2024-09-13T11:59:18.847 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:18.872 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:19.025 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 819 kB/s | 145 kB 00:00 2024-09-13T11:59:19.025 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:19.026 INFO:teuthology.orchestra.run.smithi005.stdout:Total 810 kB/s | 145 kB 00:00 2024-09-13T11:59:19.026 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:19.034 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:19.035 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:19.096 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:19.096 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:19.252 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:19.256 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:19.334 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:19.361 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:19.362 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:19.362 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:59:19.362 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:19.421 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:19.514 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:19.531 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:19.559 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:19.560 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T11:59:19.560 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:19.621 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:59:19.738 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:59:20.205 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:20.236 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:59:20.236 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:59:20.237 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:59:20.238 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:59:20.240 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:59:20.333 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:20.343 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:59:20.345 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-13T11:59:20.346 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:20.347 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 19 M 2024-09-13T11:59:20.347 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 159 M 2024-09-13T11:59:20.347 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:20.372 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:20.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:20.372 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:20.373 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:20.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:20.373 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:20.471 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:20.472 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 145 k 2024-09-13T11:59:20.473 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 432 k 2024-09-13T11:59:20.474 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:20.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr 2024-09-13T11:59:20.655 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 799 kB/s | 145 kB 00:00 2024-09-13T11:59:20.655 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:20.656 INFO:teuthology.orchestra.run.smithi039.stdout:Total 790 kB/s | 145 kB 00:00 2024-09-13T11:59:20.656 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:20.664 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:20.664 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:20.724 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:20.724 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:20.769 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 229 kB/s | 34 kB 00:00 2024-09-13T11:59:20.794 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 94 kB/s | 16 kB 00:00 2024-09-13T11:59:20.867 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:20.947 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:20.975 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:20.975 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:20.975 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:59:20.975 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:20.986 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-13T11:59:21.078 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-09-13T11:59:21.128 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.0 MB/s | 863 kB 00:00 2024-09-13T11:59:21.170 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-13T11:59:21.187 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 343 kB/s | 19 kB 00:00 2024-09-13T11:59:21.193 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:21.222 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-13T11:59:21.246 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T11:59:21.257 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:59:21.258 INFO:teuthology.orchestra.run.smithi087.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:59:21.259 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:59:21.260 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:21.261 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:21.286 INFO:teuthology.orchestra.run.smithi005.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:21.330 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T11:59:21.332 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:21.404 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:59:21.405 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 18 MB/s | 14 MB 00:00 2024-09-13T11:59:21.430 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-09-13T11:59:21.472 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.8 MB/s | 2.0 MB 00:00 2024-09-13T11:59:21.475 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:21.476 INFO:teuthology.orchestra.run.smithi037.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-13T11:59:21.524 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:21.544 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:21.545 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:21.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-09-13T11:59:21.837 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:21.838 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:21.921 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:21.922 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:21.992 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:22.132 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:59:22.133 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout:Install 12 Packages 2024-09-13T11:59:22.134 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 19 M 2024-09-13T11:59:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 159 M 2024-09-13T11:59:22.135 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:22.166 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2024-09-13T11:59:22.197 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:22.287 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:22.328 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:22.329 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T11:59:22.330 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:22.402 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-09-13T11:59:22.477 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:22.565 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:59:22.601 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:59:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:59:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:59:22.698 INFO:teuthology.orchestra.run.smithi005.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4697.gd58 106 kB/s | 16 kB 00:00 2024-09-13T11:59:22.723 INFO:teuthology.orchestra.run.smithi005.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4697.gd5 190 kB/s | 34 kB 00:00 2024-09-13T11:59:22.758 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:03 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:22.797 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:59:22.851 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:22.865 INFO:teuthology.orchestra.run.smithi005.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-13T11:59:22.895 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:22.896 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T11:59:22.896 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:22.907 INFO:teuthology.orchestra.run.smithi005.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-13T11:59:22.934 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:59:22.958 INFO:teuthology.orchestra.run.smithi005.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:59:22.963 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:22.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:59:22.983 INFO:teuthology.orchestra.run.smithi005.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-13T11:59:23.034 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:59:23.050 INFO:teuthology.orchestra.run.smithi005.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-13T11:59:23.082 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:59:23.092 INFO:teuthology.orchestra.run.smithi005.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-09-13T11:59:23.096 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:23.096 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:23.096 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:23.096 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T11:59:23.097 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:23.098 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 783 k 2024-09-13T11:59:23.098 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 789 k 2024-09-13T11:59:23.098 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:23.109 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:23.117 INFO:teuthology.orchestra.run.smithi005.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-13T11:59:23.142 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:59:23.151 INFO:teuthology.orchestra.run.smithi005.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.0 MB/s | 188 kB 00:00 2024-09-13T11:59:23.301 INFO:teuthology.orchestra.run.smithi005.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 19 MB/s | 14 MB 00:00 2024-09-13T11:59:23.326 INFO:teuthology.orchestra.run.smithi005.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 131 kB/s | 31 kB 00:00 2024-09-13T11:59:23.331 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:23.331 INFO:teuthology.orchestra.run.smithi005.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-13T11:59:23.368 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.8 MB/s | 783 kB 00:00 2024-09-13T11:59:23.369 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:23.369 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-13T11:59:23.369 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:23.374 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:23.374 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:23.380 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:23.381 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:23.381 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:23.401 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:23.401 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:23.458 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:23.545 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:04 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:23.685 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:23.686 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:23.687 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:23.688 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-09-13T11:59:23.688 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:23.688 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 19 M 2024-09-13T11:59:23.688 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 159 M 2024-09-13T11:59:23.688 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:23.692 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:23.692 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:23.930 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:24.040 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:24.266 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 235 kB/s | 34 kB 00:00 2024-09-13T11:59:24.292 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 96 kB/s | 16 kB 00:00 2024-09-13T11:59:24.301 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:24.374 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:24.411 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:59:24.417 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-13T11:59:24.455 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:59:24.500 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-09-13T11:59:24.502 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:59:24.531 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:59:24.551 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:59:24.576 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:24.612 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:24.626 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-09-13T11:59:24.645 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:59:24.685 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-09-13T11:59:24.710 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.3 MB/s | 31 kB 00:00 2024-09-13T11:59:24.735 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-09-13T11:59:24.761 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-13T11:59:24.782 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:59:24.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:59:24.881 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:59:24.928 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 18 MB/s | 14 MB 00:00 2024-09-13T11:59:24.929 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:59:24.931 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:24.931 INFO:teuthology.orchestra.run.smithi039.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-13T11:59:24.956 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:24.981 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:24.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:59:25.001 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:25.001 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:25.285 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:25.285 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:25.442 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:25.577 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:25.577 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:25.577 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-13T11:59:25.577 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 145 k 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 432 k 2024-09-13T11:59:25.578 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:25.793 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 679 kB/s | 145 kB 00:00 2024-09-13T11:59:25.793 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:25.793 INFO:teuthology.orchestra.run.smithi087.stdout:Total 674 kB/s | 145 kB 00:00 2024-09-13T11:59:25.793 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:25.802 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:25.803 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:25.858 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:25.858 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:25.899 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:25.979 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:59:26.007 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:26.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:59:26.069 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:59:26.092 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:26.107 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:59:26.118 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:26.118 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:59:26.118 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:26.212 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:59:26.347 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:59:26.447 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:59:26.496 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:59:26.523 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:26.556 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:59:26.588 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:59:26.646 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:27.034 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:27.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-09-13T11:59:27.871 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:27.962 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:28.004 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:28.005 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T11:59:28.005 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:28.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:59:28.378 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:59:28.435 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:28.628 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:28.628 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:59:28.629 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:59:28.637 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:28.770 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:28.771 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:28.771 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:28.771 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:28.771 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:28.771 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:59:28.772 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 19 M 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 159 M 2024-09-13T11:59:28.773 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:28.964 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:59:28.964 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:28.964 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:28.964 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:59:28.965 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:28.966 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:29.086 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 248 kB/s | 34 kB 00:00 2024-09-13T11:59:29.103 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 106 kB/s | 16 kB 00:00 2024-09-13T11:59:29.209 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:59:29.337 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-13T11:59:29.470 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-13T11:59:29.638 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 21 MB/s | 14 MB 00:00 2024-09-13T11:59:29.679 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.6 MB/s | 548 kB 00:00 2024-09-13T11:59:29.696 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 343 kB/s | 19 kB 00:00 2024-09-13T11:59:29.738 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-13T11:59:29.755 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-13T11:59:29.797 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 18 MB/s | 2.0 MB 00:00 2024-09-13T11:59:29.800 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:10 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:29.822 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-13T11:59:29.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:59:29.940 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:59:29.941 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:59:29.942 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================= 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:29.943 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 38 M 2024-09-13T11:59:29.944 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 212 M 2024-09-13T11:59:29.944 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:29.947 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.4 MB/s | 863 kB 00:00 2024-09-13T11:59:29.951 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:29.951 INFO:teuthology.orchestra.run.smithi087.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-13T11:59:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:29.999 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:30.019 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:30.019 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:30.308 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:30.308 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:30.361 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:59:30.384 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:59:30.385 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:59:30.436 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-13T11:59:30.461 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-09-13T11:59:30.512 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-09-13T11:59:30.570 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-13T11:59:30.687 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-13T11:59:30.696 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:59:30.696 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:30.696 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:30.696 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:59:30.697 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:59:30.698 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:30.698 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:30.887 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-13T11:59:30.894 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:30.913 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-13T11:59:30.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:59:30.965 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:59:30.980 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-09-13T11:59:31.005 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.7 MB/s | 250 kB 00:00 2024-09-13T11:59:31.007 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:59:31.055 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:59:31.085 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:59:31.097 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-13T11:59:31.195 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:59:31.328 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:59:31.429 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:59:31.490 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:59:31.526 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:31.545 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:12 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:31.558 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:59:31.580 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-13T11:59:31.581 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:31.582 INFO:teuthology.orchestra.run.smithi037.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:59:31.676 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:59:31.687 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:31.688 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-13T11:59:31.688 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:31.688 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-13T11:59:31.688 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:59:31.689 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout:======================================================================================= 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout:Install 12 Packages 2024-09-13T11:59:31.690 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:31.691 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 38 M 2024-09-13T11:59:31.691 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 212 M 2024-09-13T11:59:31.691 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:31.952 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:31.980 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:31.980 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:59:32.022 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:32.023 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:32.092 INFO:teuthology.orchestra.run.smithi005.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-13T11:59:32.134 INFO:teuthology.orchestra.run.smithi005.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 790 kB/s | 33 kB 00:00 2024-09-13T11:59:32.168 INFO:teuthology.orchestra.run.smithi005.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-13T11:59:32.218 INFO:teuthology.orchestra.run.smithi005.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-13T11:59:32.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:59:32.260 INFO:teuthology.orchestra.run.smithi005.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 927 kB/s | 38 kB 00:00 2024-09-13T11:59:32.344 INFO:teuthology.orchestra.run.smithi005.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-09-13T11:59:32.394 INFO:teuthology.orchestra.run.smithi005.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-13T11:59:32.437 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:32.437 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:32.486 INFO:teuthology.orchestra.run.smithi005.stdout:(8/12): ceph-mgr-diskprediction-local-19.3.0-46 15 MB/s | 7.4 MB 00:00 2024-09-13T11:59:32.611 INFO:teuthology.orchestra.run.smithi005.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2024-09-13T11:59:32.695 INFO:teuthology.orchestra.run.smithi005.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-13T11:59:32.720 INFO:teuthology.orchestra.run.smithi005.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-09-13T11:59:32.830 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:13 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:32.968 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:32.969 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-13T11:59:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:59:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================= 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2024-09-13T11:59:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:32.972 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 38 M 2024-09-13T11:59:32.972 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 212 M 2024-09-13T11:59:32.972 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:33.104 INFO:teuthology.orchestra.run.smithi005.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-13T11:59:33.104 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:33.105 INFO:teuthology.orchestra.run.smithi005.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-13T11:59:33.243 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:33.313 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:59:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:33.434 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-13T11:59:33.458 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:33.470 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:33.484 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 658 kB/s | 33 kB 00:00 2024-09-13T11:59:33.497 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:59:33.497 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:33.497 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:33.634 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.0 MB/s | 801 kB 00:00 2024-09-13T11:59:33.689 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:59:33.718 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-46 16 MB/s | 7.4 MB 00:00 2024-09-13T11:59:33.720 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:59:33.734 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 182 kB/s | 18 kB 00:00 2024-09-13T11:59:33.799 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:59:33.818 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 385 kB/s | 38 kB 00:00 2024-09-13T11:59:33.951 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:33.951 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:33.952 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-13T11:59:34.477 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-09-13T11:59:34.486 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:59:34.586 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-13T11:59:34.603 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:59:34.728 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:59:34.736 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:34.788 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:34.811 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:59:34.853 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:34.957 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:34.994 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.9 MB/s | 4.9 MB 00:01 2024-09-13T11:59:34.995 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:59:35.187 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:59:35.219 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:59:35.323 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:59:35.503 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.3 MB/s | 5.1 MB 00:01 2024-09-13T11:59:35.918 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:36.016 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:59:36.136 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:59:36.683 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:59:36.683 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:59:36.683 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:59:36.683 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:59:36.684 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:36.996 INFO:teuthology.orchestra.run.smithi087.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:36.997 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:37.000 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:59:37.027 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:37.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:59:37.435 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:37.719 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.2 MB/s | 16 MB 00:03 2024-09-13T11:59:37.720 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:37.721 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.0 MB/s | 38 MB 00:04 2024-09-13T11:59:37.829 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:08 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:37.962 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:59:37.963 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2024-09-13T11:59:37.964 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2024-09-13T11:59:37.965 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:37.965 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 38 M 2024-09-13T11:59:37.965 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 212 M 2024-09-13T11:59:37.965 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:38.091 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:38.118 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:38.118 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:38.517 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:59:38.543 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:38.579 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:38.579 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:38.811 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-46 17 MB/s | 7.4 MB 00:00 2024-09-13T11:59:38.878 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 494 kB/s | 33 kB 00:00 2024-09-13T11:59:39.070 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-09-13T11:59:39.112 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 441 kB/s | 18 kB 00:00 2024-09-13T11:59:39.154 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 925 kB/s | 38 kB 00:00 2024-09-13T11:59:39.187 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): libquadmath-11.5.0-2.el9.x86_64.rpm 234 kB/s | 191 kB 00:00 2024-09-13T11:59:39.304 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 33 MB/s | 4.9 MB 00:00 2024-09-13T11:59:39.346 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 820 kB/s | 801 kB 00:00 2024-09-13T11:59:39.363 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:39.421 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-13T11:59:39.441 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:59:39.472 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:39.473 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:59:39.484 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:39.488 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-13T11:59:39.547 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.2 MB/s | 250 kB 00:00 2024-09-13T11:59:39.589 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:39.627 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:59:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:59:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:59:39.929 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:59:39.931 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:39.932 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:40.239 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-13T11:59:40.240 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:40.240 INFO:teuthology.orchestra.run.smithi087.stdout:Total 17 MB/s | 38 MB 00:02 2024-09-13T11:59:40.278 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:59:40.591 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:40.617 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:40.618 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:40.644 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:59:40.768 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:59:40.874 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:21 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:40.974 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:40.974 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:40.975 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:59:41.013 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:59:41.015 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-09-13T11:59:41.016 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:41.017 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.2 M 2024-09-13T11:59:41.017 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-09-13T11:59:41.017 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:41.036 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:41.036 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:41.425 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:59:41.426 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:59:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:59:41.427 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:41.427 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:41.475 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 284 kB/s | 49 kB 00:00 2024-09-13T11:59:41.593 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 88 kB/s | 26 kB 00:00 2024-09-13T11:59:41.618 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 61 kB/s | 19 kB 00:00 2024-09-13T11:59:41.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:59:41.802 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:41.884 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:59:41.920 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:41.993 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 424 kB/s | 159 kB 00:00 2024-09-13T11:59:42.022 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:42.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:59:42.071 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:42.077 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 608 kB/s | 279 kB 00:00 2024-09-13T11:59:42.102 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 355 kB/s | 222 kB 00:00 2024-09-13T11:59:42.127 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 401 kB/s | 54 kB 00:00 2024-09-13T11:59:42.177 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-09-13T11:59:42.202 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-13T11:59:42.228 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T11:59:42.246 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:59:42.253 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-13T11:59:42.278 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-13T11:59:42.278 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:59:42.312 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-13T11:59:42.315 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:42.315 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-13T11:59:42.355 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:59:42.386 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:42.403 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:42.403 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:42.410 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:23 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:42.500 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:42.500 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:42.557 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:59:42.559 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:42.560 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:42.561 INFO:teuthology.orchestra.run.smithi005.stdout:Install 13 Packages 2024-09-13T11:59:42.561 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:42.561 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.2 M 2024-09-13T11:59:42.561 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 26 M 2024-09-13T11:59:42.561 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:42.928 INFO:teuthology.orchestra.run.smithi005.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 310 kB/s | 26 kB 00:00 2024-09-13T11:59:42.953 INFO:teuthology.orchestra.run.smithi005.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 177 kB/s | 19 kB 00:00 2024-09-13T11:59:43.012 INFO:teuthology.orchestra.run.smithi005.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-13T11:59:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:59:43.024 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:43.037 INFO:teuthology.orchestra.run.smithi005.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 159 kB 00:00 2024-09-13T11:59:43.063 INFO:teuthology.orchestra.run.smithi005.stdout:(5/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 226 kB/s | 49 kB 00:00 2024-09-13T11:59:43.088 INFO:teuthology.orchestra.run.smithi005.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T11:59:43.113 INFO:teuthology.orchestra.run.smithi005.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-13T11:59:43.134 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:59:43.144 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:59:43.163 INFO:teuthology.orchestra.run.smithi005.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-13T11:59:43.166 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:43.189 INFO:teuthology.orchestra.run.smithi005.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-13T11:59:43.192 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:59:43.214 INFO:teuthology.orchestra.run.smithi005.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T11:59:43.239 INFO:teuthology.orchestra.run.smithi005.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-13T11:59:43.256 INFO:teuthology.orchestra.run.smithi005.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-13T11:59:43.256 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:59:43.290 INFO:teuthology.orchestra.run.smithi005.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.2 MB/s | 1.0 MB 00:00 2024-09-13T11:59:43.293 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:43.293 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-13T11:59:43.320 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:59:43.356 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:59:43.362 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:43.378 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:43.379 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:43.428 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:59:43.476 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:43.476 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:43.538 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:59:43.665 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:59:43.716 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:59:44.129 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:44.304 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:59:44.368 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:59:44.409 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:44.433 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:59:44.468 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:59:44.480 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:59:44.510 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:59:44.529 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:59:44.563 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:59:44.612 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:59:44.616 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:59:44.641 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:44.727 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:59:44.795 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:59:45.436 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:59:45.446 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:45.446 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:59:45.446 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:59:45.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:59:45.469 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:45.572 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:59:45.617 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:59:45.650 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:59:45.699 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:59:45.719 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:59:45.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:59:45.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:59:45.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:59:45.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:45.720 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:59:45.728 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:45.787 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:59:45.788 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:59:45.789 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:59:45.789 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:59:45.789 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:59:45.789 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:45.789 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:46.033 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:59:46.146 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:46.147 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:59:46.476 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:59:46.477 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:59:46.477 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:59:46.477 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:59:46.504 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:59:46.625 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:46.764 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:59:46.765 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:46.766 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 7.0 M 2024-09-13T11:59:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-09-13T11:59:46.767 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:59:46.810 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:46.811 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:47.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:59:47.099 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:47.238 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:59:47.239 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:47.240 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2024-09-13T11:59:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:47.241 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.2 M 2024-09-13T11:59:47.241 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2024-09-13T11:59:47.241 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:47.467 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 287 kB/s | 49 kB 00:00 2024-09-13T11:59:47.480 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 713 kB/s | 154 kB 00:00 2024-09-13T11:59:47.530 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:59:47.556 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T11:59:47.589 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-13T11:59:47.630 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:47.651 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 72 kB/s | 26 kB 00:00 2024-09-13T11:59:47.684 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 49 kB/s | 19 kB 00:00 2024-09-13T11:59:47.769 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:47.769 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:59:47.770 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout:Install 6 Packages 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 7.0 M 2024-09-13T11:59:47.771 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 33 M 2024-09-13T11:59:47.772 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:47.815 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 451 kB/s | 248 kB 00:00 2024-09-13T11:59:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:59:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:59:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:59:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:59:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:59:48.143 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 346 kB/s | 159 kB 00:00 2024-09-13T11:59:48.248 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.1 MB/s | 6.0 MB 00:00 2024-09-13T11:59:48.250 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:48.251 INFO:teuthology.orchestra.run.smithi037.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-13T11:59:48.251 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 283 kB/s | 222 kB 00:00 2024-09-13T11:59:48.277 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 471 kB/s | 279 kB 00:00 2024-09-13T11:59:48.302 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 338 kB/s | 54 kB 00:00 2024-09-13T11:59:48.326 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:48.352 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-13T11:59:48.377 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-13T11:59:48.403 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T11:59:48.428 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-13T11:59:48.445 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:48.445 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:48.453 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T11:59:48.465 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:59:48.466 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:48.467 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:59:48.467 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:59:48.467 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:48.467 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:48.486 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-13T11:59:48.490 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:48.490 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-13T11:59:48.560 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:48.577 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:48.578 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:48.626 INFO:teuthology.orchestra.run.smithi005.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 672 kB/s | 154 kB 00:00 2024-09-13T11:59:48.674 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:48.676 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:48.692 INFO:teuthology.orchestra.run.smithi005.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:59:48.718 INFO:teuthology.orchestra.run.smithi005.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T11:59:48.743 INFO:teuthology.orchestra.run.smithi005.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-13T11:59:48.764 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:48.932 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:59:48.938 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:59:49.001 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:59:49.176 INFO:teuthology.orchestra.run.smithi005.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 318 kB/s | 248 kB 00:00 2024-09-13T11:59:49.193 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:49.424 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:59:49.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:59:49.511 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:20 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:49.536 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:59:49.551 INFO:teuthology.orchestra.run.smithi005.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-13T11:59:49.554 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:49.555 INFO:teuthology.orchestra.run.smithi005.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-09-13T11:59:49.556 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:59:49.572 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:59:49.615 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:59:49.631 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:59:49.643 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:49.644 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:49.645 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:49.645 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:49.645 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:49.645 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:49.645 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:59:49.646 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.2 M 2024-09-13T11:59:49.647 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2024-09-13T11:59:49.648 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:49.715 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:59:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:59:49.745 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:49.753 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:49.754 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:49.831 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:59:49.884 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:59:50.198 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:50.378 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 159 kB/s | 19 kB 00:00 2024-09-13T11:59:50.379 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:59:50.403 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 176 kB/s | 26 kB 00:00 2024-09-13T11:59:50.429 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 288 kB/s | 49 kB 00:00 2024-09-13T11:59:50.430 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:59:50.537 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-13T11:59:50.545 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:50.545 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:59:50.545 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:59:50.545 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:59:50.546 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:59:50.546 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:59:50.612 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 760 kB/s | 159 kB 00:00 2024-09-13T11:59:50.629 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 585 kB/s | 54 kB 00:00 2024-09-13T11:59:50.641 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:59:50.663 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 643 kB/s | 32 kB 00:00 2024-09-13T11:59:50.688 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 242 kB/s | 14 kB 00:00 2024-09-13T11:59:50.696 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:59:50.713 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.5 MB/s | 220 kB 00:00 2024-09-13T11:59:50.730 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:59:50.730 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-13T11:59:50.755 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 853 kB/s | 279 kB 00:00 2024-09-13T11:59:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:59:50.781 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T11:59:50.808 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:50.822 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-13T11:59:50.825 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:50.825 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-13T11:59:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:50.837 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:50.895 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:50.911 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:50.911 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:50.995 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:59:50.999 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:51.000 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:51.037 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-09-13T11:59:51.089 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:59:51.171 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:59:51.199 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:51.516 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:51.569 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:51.569 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:59:51.570 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:59:51.632 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:51.678 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:59:51.740 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:59:51.772 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:51.773 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:51.773 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:51.773 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:51.774 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:51.775 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:51.775 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-09-13T11:59:51.776 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:51.776 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 979 k 2024-09-13T11:59:51.776 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.9 M 2024-09-13T11:59:51.776 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:51.804 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:59:51.841 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:59:51.883 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:59:51.918 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:59:51.919 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:59:51.920 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:59:51.920 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:59:51.920 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:59:51.920 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:51.920 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:51.924 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 617 kB/s | 80 kB 00:00 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:59:51.940 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:59:51.993 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:59:52.066 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T11:59:52.067 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:52.067 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.3 MB/s | 979 kB 00:00 2024-09-13T11:59:52.079 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:52.102 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:52.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:59:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:59:52.168 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:59:52.187 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:52.188 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:52.251 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:59:52.251 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:52.252 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:52.388 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:52.524 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-fuse 2024-09-13T11:59:52.532 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:59:52.611 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:52.700 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:52.841 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:59:52.842 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2024-09-13T11:59:52.843 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:52.844 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 7.0 M 2024-09-13T11:59:52.844 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 33 M 2024-09-13T11:59:52.844 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:59:52.933 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:59:52.965 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:59:53.013 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:59:53.043 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:53.130 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:53.269 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout:Install 2 Packages 2024-09-13T11:59:53.270 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:53.271 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 979 k 2024-09-13T11:59:53.271 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 2.9 M 2024-09-13T11:59:53.271 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:53.305 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:59:53.364 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-09-13T11:59:53.389 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 575 kB/s | 154 kB 00:00 2024-09-13T11:59:53.414 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 846 kB/s | 248 kB 00:00 2024-09-13T11:59:53.438 INFO:teuthology.orchestra.run.smithi005.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 681 kB/s | 80 kB 00:00 2024-09-13T11:59:53.464 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-13T11:59:53.489 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-13T11:59:53.523 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-13T11:59:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:53.526 INFO:teuthology.orchestra.run.smithi039.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-13T11:59:53.579 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:59:53.579 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:53.580 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:53.580 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:59:53.580 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:53.580 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:53.596 INFO:teuthology.orchestra.run.smithi005.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T11:59:53.597 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:53.597 INFO:teuthology.orchestra.run.smithi005.stdout:Total 2.9 MB/s | 979 kB 00:00 2024-09-13T11:59:53.598 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:53.609 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:53.610 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:53.610 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:53.632 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:53.632 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:53.716 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:53.717 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:53.718 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:53.719 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:53.777 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:59:53.778 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:59:53.779 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:59:53.779 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:59:53.779 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:59:53.779 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:59:53.779 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:59:53.832 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-volume 2024-09-13T11:59:53.914 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:54.042 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:54.061 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:59:54.136 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:54.137 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:54.141 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:59:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:59:54.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:59:54.435 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:59:54.575 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 285 k 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.3 M 2024-09-13T11:59:54.576 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:54.795 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-13T11:59:54.796 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:54.796 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-13T11:59:54.797 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:54.806 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:54.806 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:54.814 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:54.814 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:59:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:59:54.840 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:54.840 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:59:54.959 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:25 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:54.998 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:59:55.028 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:55.072 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:55.091 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:59:55.092 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:55.093 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-09-13T11:59:55.094 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:55.095 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 7.0 M 2024-09-13T11:59:55.095 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2024-09-13T11:59:55.095 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:55.147 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:55.164 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:55.164 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:55.164 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:59:55.164 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:55.343 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-volume 2024-09-13T11:59:55.616 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 719 kB/s | 154 kB 00:00 2024-09-13T11:59:55.666 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:59:55.699 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 833 kB/s | 248 kB 00:00 2024-09-13T11:59:55.716 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T11:59:55.800 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-13T11:59:55.825 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 594 kB/s | 74 kB 00:00 2024-09-13T11:59:55.827 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:55.827 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.6 MB/s | 7.0 MB 00:00 2024-09-13T11:59:55.867 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:55.868 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:59:55.868 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:59:55.868 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:59:55.868 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:59:55.868 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:59:55.898 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:55.910 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:55.910 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:55.937 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:56.010 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:56.012 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:56.079 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:56.080 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T11:59:56.081 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:56.081 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 285 k 2024-09-13T11:59:56.081 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 1.3 M 2024-09-13T11:59:56.081 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:59:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:56.143 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:56.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:56.160 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:56.161 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:56.161 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:56.293 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-13T11:59:56.294 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:56.294 INFO:teuthology.orchestra.run.smithi005.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-13T11:59:56.294 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:56.303 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:56.304 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T11:59:56.335 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:56.336 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:56.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2024-09-13T11:59:56.426 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-09-13T11:59:56.465 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:59:56.527 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:59:56.574 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:56.654 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:56.671 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:56.671 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:59:56.672 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:59:56.672 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:56.993 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:57.022 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:57.062 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:59:57.133 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T11:59:57.133 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2024-09-13T11:59:57.134 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:57.135 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 979 k 2024-09-13T11:59:57.135 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 2.9 M 2024-09-13T11:59:57.135 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T11:59:57.147 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:59:57.161 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:57.162 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T11:59:57.163 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T11:59:57.163 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T11:59:57.163 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:57.164 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 127 k 2024-09-13T11:59:57.164 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-09-13T11:59:57.164 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T11:59:57.213 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:59:57.248 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:57.350 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 678 kB/s | 127 kB 00:00 2024-09-13T11:59:57.351 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:57.351 INFO:teuthology.orchestra.run.smithi037.stdout:Total 672 kB/s | 127 kB 00:00 2024-09-13T11:59:57.351 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T11:59:57.357 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T11:59:57.357 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T11:59:57.426 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T11:59:57.426 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T11:59:57.433 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T11:59:57.579 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T11:59:57.658 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 159 kB/s | 80 kB 00:00 2024-09-13T11:59:57.659 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:57.659 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.8 MB/s | 979 kB 00:00 2024-09-13T11:59:57.671 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T11:59:57.684 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:59:57.693 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T11:59:57.694 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T11:59:57.749 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:59:57.749 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:57.750 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T11:59:57.750 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:57.750 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:57.750 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T11:59:57.779 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T11:59:57.779 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T11:59:57.974 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T11:59:58.034 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados-devel 2024-09-13T11:59:58.089 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:59:58.089 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:59:58.089 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:59:58.089 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:59:58.090 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:59:58.090 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:59:58.123 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:59:58.128 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:59:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:59:58.373 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:59:58.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:59:58.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:59:58.374 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:59:58.374 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:58.374 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T11:59:58.402 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:58.611 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-09-13T11:59:58.630 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-09-13T11:59:58.630 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:58.768 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T11:59:58.769 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T11:59:58.770 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T11:59:58.770 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T11:59:58.770 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 127 k 2024-09-13T11:59:58.770 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 456 k 2024-09-13T11:59:58.770 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T11:59:58.847 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:59:58.848 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:59:58.982 INFO:teuthology.orchestra.run.smithi005.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 596 kB/s | 127 kB 00:00 2024-09-13T11:59:58.983 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:58.983 INFO:teuthology.orchestra.run.smithi005.stdout:Total 592 kB/s | 127 kB 00:00 2024-09-13T11:59:58.983 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T11:59:58.989 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T11:59:58.989 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T11:59:59.058 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T11:59:59.058 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T11:59:59.142 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:59:59.142 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:59.143 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T11:59:59.143 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:59:59.143 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T11:59:59.143 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T11:59:59.198 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T11:59:59.205 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:59.219 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T11:59:59.299 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:59:59.325 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T11:59:59.330 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T11:59:59.331 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 979 k 2024-09-13T11:59:59.332 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.9 M 2024-09-13T11:59:59.332 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T11:59:59.344 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T11:59:59.346 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T11:59:59.346 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T11:59:59.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-volume 2024-09-13T11:59:59.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-09-13T11:59:59.610 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 537 kB/s | 80 kB 00:00 2024-09-13T11:59:59.752 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-13T11:59:59.752 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:59:59.753 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-13T11:59:59.759 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:59:59.764 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T11:59:59.785 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T11:59:59.786 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T11:59:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T11:59:59.867 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T11:59:59.984 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T11:59:59.998 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:00.033 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:00.033 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:00.033 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T12:00:00.033 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:00.034 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:00.034 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:00.055 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:00.124 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:00.124 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:00.124 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 285 k 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.3 M 2024-09-13T12:00:00.125 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:00.138 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T12:00:00.139 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:00.140 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 32 k 2024-09-13T12:00:00.140 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 142 k 2024-09-13T12:00:00.140 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T12:00:00.197 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T12:00:00.284 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T12:00:00.286 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-13T12:00:00.286 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:00.287 INFO:teuthology.orchestra.run.smithi037.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-13T12:00:00.287 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T12:00:00.291 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T12:00:00.291 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T12:00:00.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs2 2024-09-13T12:00:00.312 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T12:00:00.313 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T12:00:00.363 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T12:00:00.363 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:00.364 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T12:00:00.364 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:00.373 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:00.373 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:00.406 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T12:00:00.426 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:00.426 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:00.520 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:00.657 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:00.715 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:00.732 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:00.732 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:00.732 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T12:00:00.733 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:00.891 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:00.911 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:00.985 INFO:teuthology.orchestra.run.smithi005.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:01.028 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T12:00:01.029 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T12:00:01.031 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:01.032 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:01.033 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:01.096 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install libcephfs-devel 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:01.160 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:01.298 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T12:00:01.298 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:01.298 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:01.298 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T12:00:01.298 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:01.299 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:01.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-09-13T12:00:01.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2024-09-13T12:00:01.688 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T12:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:01.828 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T12:00:01.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:01.830 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 32 k 2024-09-13T12:00:01.830 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 142 k 2024-09-13T12:00:01.830 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T12:00:01.910 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:01.970 INFO:teuthology.orchestra.run.smithi005.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 227 kB/s | 32 kB 00:00 2024-09-13T12:00:01.970 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:01.970 INFO:teuthology.orchestra.run.smithi005.stdout:Total 225 kB/s | 32 kB 00:00 2024-09-13T12:00:01.971 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T12:00:01.974 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T12:00:01.975 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T12:00:01.990 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T12:00:01.990 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T12:00:02.005 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:02.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2024-09-13T12:00:02.050 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:02.051 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:02.051 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:02.087 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T12:00:02.108 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-09-13T12:00:02.152 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:02.195 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:02.290 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:02.291 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:02.291 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-09-13T12:00:02.291 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:02.291 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:02.291 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 285 k 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.3 M 2024-09-13T12:00:02.292 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:02.515 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T12:00:02.516 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:02.516 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T12:00:02.516 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:02.525 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:02.525 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:02.557 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:02.557 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:02.588 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:02.636 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:02.700 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:02.777 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 127 k 2024-09-13T12:00:02.778 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 456 k 2024-09-13T12:00:02.779 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:02.782 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:02.793 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:02.832 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:02.840 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:02.842 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:02.842 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:02.850 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:02.866 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:02.866 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:02.866 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T12:00:02.867 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:02.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-09-13T12:00:02.976 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 642 kB/s | 127 kB 00:00 2024-09-13T12:00:02.976 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:02.977 INFO:teuthology.orchestra.run.smithi039.stdout:Total 637 kB/s | 127 kB 00:00 2024-09-13T12:00:02.977 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:02.983 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:02.983 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:03.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librados2 2024-09-13T12:00:03.050 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:03.050 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:03.206 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:03.318 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:03.494 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:03.587 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:03.609 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:03.632 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:03.634 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:03.634 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:03.691 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-09-13T12:00:03.702 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:03.748 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:03.750 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:03.751 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:03.761 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:03.823 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install librbd1 2024-09-13T12:00:03.872 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T12:00:03.872 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:03.872 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:03.872 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T12:00:03.873 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:03.873 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:04.000 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:04.221 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-09-13T12:00:04.254 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2024-09-13T12:00:04.285 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:04.377 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:04.414 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:04.423 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:04.424 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:04.424 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:04.483 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-09-13T12:00:04.509 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:04.554 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:04.555 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:04.555 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:04.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rados 2024-09-13T12:00:04.799 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:04.850 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:04.934 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:04.935 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:04.936 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:04.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:04.936 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 127 k 2024-09-13T12:00:04.936 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2024-09-13T12:00:04.936 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:04.942 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:04.989 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:04.990 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:04.990 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:05.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2024-09-13T12:00:05.075 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:05.120 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 688 kB/s | 127 kB 00:00 2024-09-13T12:00:05.121 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:05.121 INFO:teuthology.orchestra.run.smithi087.stdout:Total 682 kB/s | 127 kB 00:00 2024-09-13T12:00:05.121 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:05.126 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:05.126 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:05.168 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:05.190 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:05.190 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:05.206 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:05.214 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:05.215 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:05.215 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:05.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-09-13T12:00:05.299 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:05.345 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:05.347 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:05.347 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:05.351 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:05.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rgw 2024-09-13T12:00:05.467 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:05.643 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:05.783 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:05.783 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:05.783 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-13T12:00:05.783 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:05.784 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:05.785 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 32 k 2024-09-13T12:00:05.785 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 142 k 2024-09-13T12:00:05.785 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:05.876 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:05.912 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:05.946 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 196 kB/s | 32 kB 00:00 2024-09-13T12:00:05.947 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:05.947 INFO:teuthology.orchestra.run.smithi039.stdout:Total 195 kB/s | 32 kB 00:00 2024-09-13T12:00:05.947 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:05.951 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:05.951 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:05.966 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:05.966 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:05.968 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:06.007 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:06.014 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:06.015 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-09-13T12:00:06.016 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:06.058 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:06.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-09-13T12:00:06.099 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:06.145 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:06.146 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:06.146 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:06.154 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:06.175 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:06.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:06.176 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:06.176 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:06.176 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:06.176 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:06.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-cephfs 2024-09-13T12:00:06.413 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-09-13T12:00:06.539 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:06.664 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:06.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:06.832 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:06.833 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:06.833 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:06.833 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:06.833 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:06.834 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:06.834 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:06.834 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 85 k 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 231 k 2024-09-13T12:00:06.835 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T12:00:06.899 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:06.944 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:06.945 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:06.945 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:06.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2024-09-13T12:00:06.997 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:06.998 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 444 kB/s | 85 kB 00:00 2024-09-13T12:00:06.998 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:06.999 INFO:teuthology.orchestra.run.smithi037.stdout:Total 441 kB/s | 85 kB 00:00 2024-09-13T12:00:06.999 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T12:00:07.006 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T12:00:07.006 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T12:00:07.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install python3-rbd 2024-09-13T12:00:07.074 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T12:00:07.075 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T12:00:07.088 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:07.132 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:07.134 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:07.134 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:07.224 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T12:00:07.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-09-13T12:00:07.332 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:07.554 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:07.612 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:07.648 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:07.693 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:07.694 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:07.694 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:07.704 INFO:teuthology.orchestra.run.smithi005.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:07.709 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:07.750 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:07.751 INFO:teuthology.orchestra.run.smithi005.stdout:Nothing to do. 2024-09-13T12:00:07.752 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:07.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2024-09-13T12:00:07.793 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:07.821 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-fuse 2024-09-13T12:00:07.925 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:07.926 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:07.926 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-13T12:00:07.926 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 k 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 142 k 2024-09-13T12:00:07.927 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:07.951 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T12:00:07.951 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:07.952 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:07.952 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:08.090 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 195 kB/s | 32 kB 00:00 2024-09-13T12:00:08.090 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:08.091 INFO:teuthology.orchestra.run.smithi087.stdout:Total 194 kB/s | 32 kB 00:00 2024-09-13T12:00:08.091 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:08.094 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:08.094 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:08.117 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:08.117 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:08.208 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:08.226 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-09-13T12:00:08.311 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:08.343 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:08.412 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:08.436 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:08.483 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:08.484 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:08.484 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:08.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2024-09-13T12:00:08.552 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:08.552 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:08.552 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:08.552 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:08.552 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:08.553 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 85 k 2024-09-13T12:00:08.554 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 231 k 2024-09-13T12:00:08.554 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T12:00:08.694 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:08.723 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 502 kB/s | 85 kB 00:00 2024-09-13T12:00:08.724 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:08.724 INFO:teuthology.orchestra.run.smithi005.stdout:Total 498 kB/s | 85 kB 00:00 2024-09-13T12:00:08.724 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T12:00:08.732 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T12:00:08.732 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T12:00:08.803 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T12:00:08.803 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T12:00:08.828 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:08.911 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:08.911 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:08.911 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:08.911 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:08.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:08.912 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:08.953 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T12:00:08.970 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:08.971 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T12:00:08.972 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:08.972 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.2 M 2024-09-13T12:00:08.972 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 14 M 2024-09-13T12:00:08.972 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T12:00:09.065 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:09.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-09-13T12:00:09.135 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:09.227 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:09.274 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:09.275 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:09.275 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:09.317 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-13T12:00:09.317 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:09.318 INFO:teuthology.orchestra.run.smithi037.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-13T12:00:09.318 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T12:00:09.328 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T12:00:09.328 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T12:00:09.340 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2024-09-13T12:00:09.420 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T12:00:09.420 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T12:00:09.462 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:09.656 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:09.714 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:09.714 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:09.714 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T12:00:09.714 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:09.715 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:09.715 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:09.746 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:09.790 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:09.791 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:09.791 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:09.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-09-13T12:00:09.895 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T12:00:09.934 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:09.951 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:09.978 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:10.026 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:10.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-mirror 2024-09-13T12:00:10.072 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:10.073 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:10.073 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:10.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2024-09-13T12:00:10.426 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:10.517 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:10.560 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:10.561 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:10.561 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:10.629 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:10.635 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-09-13T12:00:10.730 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:10.770 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:10.770 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repo Size 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T12:00:10.771 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:10.772 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 3.2 M 2024-09-13T12:00:10.772 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 14 M 2024-09-13T12:00:10.772 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T12:00:10.824 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:10.869 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:10.870 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:10.871 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:10.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2024-09-13T12:00:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:11.108 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T12:00:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:11.109 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:11.168 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 8.0 MB/s | 3.2 MB 00:00 2024-09-13T12:00:11.169 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:11.169 INFO:teuthology.orchestra.run.smithi005.stdout:Total 8.0 MB/s | 3.2 MB 00:00 2024-09-13T12:00:11.169 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T12:00:11.179 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T12:00:11.179 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T12:00:11.195 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:11.271 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T12:00:11.271 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T12:00:11.283 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:11.323 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-09-13T12:00:11.325 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:11.327 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:11.327 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:11.397 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-09-13T12:00:11.532 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:11.624 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:11.669 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:11.670 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2024-09-13T12:00:11.671 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:11.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2024-09-13T12:00:11.748 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T12:00:11.794 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:11.821 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:11.923 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:11.965 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:12.056 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:12.061 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T12:00:12.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 170 k 2024-09-13T12:00:12.063 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 490 k 2024-09-13T12:00:12.064 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-09-13T12:00:12.099 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:12.100 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:12.100 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:12.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-09-13T12:00:12.278 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 794 kB/s | 170 kB 00:00 2024-09-13T12:00:12.279 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:12.279 INFO:teuthology.orchestra.run.smithi037.stdout:Total 787 kB/s | 170 kB 00:00 2024-09-13T12:00:12.279 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-09-13T12:00:12.288 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-09-13T12:00:12.288 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-09-13T12:00:12.330 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:12.356 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-09-13T12:00:12.356 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-09-13T12:00:12.470 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:12.471 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 85 k 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 231 k 2024-09-13T12:00:12.472 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:12.514 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-09-13T12:00:12.621 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:12.707 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 363 kB/s | 85 kB 00:00 2024-09-13T12:00:12.708 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:12.708 INFO:teuthology.orchestra.run.smithi039.stdout:Total 361 kB/s | 85 kB 00:00 2024-09-13T12:00:12.708 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:12.715 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:12.717 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:12.738 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:12.782 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:12.783 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:12.829 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:12.872 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:12.873 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:12.873 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:12.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-09-13T12:00:13.040 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.049 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:13.081 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:13.165 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.298 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install rbd-nbd 2024-09-13T12:00:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:13.322 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-09-13T12:00:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:13.323 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-09-13T12:00:13.507 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:13.551 DEBUG:teuthology.parallel:result is None 2024-09-13T12:00:13.597 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T12:00:13.640 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:13.641 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-09-13T12:00:13.641 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:13.687 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-09-13T12:00:13.911 INFO:teuthology.orchestra.run.smithi005.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:13.958 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:14.050 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:14.051 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:Install 1 Package 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 170 k 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 490 k 2024-09-13T12:00:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-09-13T12:00:14.217 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2024-09-13T12:00:14.256 INFO:teuthology.orchestra.run.smithi005.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 835 kB/s | 170 kB 00:00 2024-09-13T12:00:14.257 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:14.257 INFO:teuthology.orchestra.run.smithi005.stdout:Total 829 kB/s | 170 kB 00:00 2024-09-13T12:00:14.257 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-09-13T12:00:14.266 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-09-13T12:00:14.266 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-09-13T12:00:14.281 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:14.333 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-09-13T12:00:14.333 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-09-13T12:00:14.417 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:14.417 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:14.417 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:14.417 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:14.417 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2024-09-13T12:00:14.418 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:14.499 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-09-13T12:00:14.599 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:14.609 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 445 kB/s | 85 kB 00:00 2024-09-13T12:00:14.610 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:14.610 INFO:teuthology.orchestra.run.smithi087.stdout:Total 441 kB/s | 85 kB 00:00 2024-09-13T12:00:14.610 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:14.617 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:14.617 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:14.685 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:14.685 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:14.826 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:14.837 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:14.947 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:14.967 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:14.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:14.969 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.2 M 2024-09-13T12:00:14.969 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 14 M 2024-09-13T12:00:14.969 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:15.042 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:15.339 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:15.339 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:15.339 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-09-13T12:00:15.339 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:15.340 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:15.340 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-09-13T12:00:15.345 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:15.370 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 7.9 MB/s | 3.2 MB 00:00 2024-09-13T12:00:15.370 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:15.371 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.9 MB/s | 3.2 MB 00:00 2024-09-13T12:00:15.371 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:15.381 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:15.382 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:15.474 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:15.474 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:15.594 DEBUG:teuthology.parallel:result is None 2024-09-13T12:00:15.598 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:15.598 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:15.598 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:15.598 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:15.599 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:15.599 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:15.879 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-09-13T12:00:16.030 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:16.087 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:16.115 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:16.480 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:16.616 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:16.616 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:16.616 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.2 M 2024-09-13T12:00:16.617 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 14 M 2024-09-13T12:00:16.618 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:16.997 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 8.4 MB/s | 3.2 MB 00:00 2024-09-13T12:00:16.997 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:16.998 INFO:teuthology.orchestra.run.smithi087.stdout:Total 8.3 MB/s | 3.2 MB 00:00 2024-09-13T12:00:16.998 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:17.006 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:17.007 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:17.092 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:17.092 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:17.223 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:17.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2024-09-13T12:00:17.553 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:17.614 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T12:00:17.640 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:18.025 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:59 ago on Fri 13 Sep 2024 11:58:19 AM UTC. 2024-09-13T12:00:18.164 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2024-09-13T12:00:18.165 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:18.166 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 170 k 2024-09-13T12:00:18.166 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 490 k 2024-09-13T12:00:18.166 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-09-13T12:00:18.372 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 826 kB/s | 170 kB 00:00 2024-09-13T12:00:18.373 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:18.373 INFO:teuthology.orchestra.run.smithi039.stdout:Total 820 kB/s | 170 kB 00:00 2024-09-13T12:00:18.373 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-09-13T12:00:18.382 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-09-13T12:00:18.382 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-09-13T12:00:18.450 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-09-13T12:00:18.450 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-09-13T12:00:18.608 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-09-13T12:00:18.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:18.703 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:18.703 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:18.706 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:18.932 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-09-13T12:00:19.170 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:19.423 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:19.423 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:19.423 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-09-13T12:00:19.423 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:19.423 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:19.424 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-09-13T12:00:19.512 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:58:29 AM UTC. 2024-09-13T12:00:19.651 DEBUG:teuthology.parallel:result is None 2024-09-13T12:00:19.652 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-09-13T12:00:19.652 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:19.652 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-09-13T12:00:19.652 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:19.652 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 170 k 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 490 k 2024-09-13T12:00:19.653 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-09-13T12:00:19.866 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 791 kB/s | 170 kB 00:00 2024-09-13T12:00:19.867 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-09-13T12:00:19.867 INFO:teuthology.orchestra.run.smithi087.stdout:Total 786 kB/s | 170 kB 00:00 2024-09-13T12:00:19.867 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-09-13T12:00:19.875 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-09-13T12:00:19.875 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-09-13T12:00:19.939 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-09-13T12:00:19.940 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-09-13T12:00:20.096 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-09-13T12:00:20.206 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:20.640 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:20.923 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-09-13T12:00:21.172 DEBUG:teuthology.parallel:result is None 2024-09-13T12:00:21.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T12:00:21.326 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:00:21.353 INFO:teuthology.orchestra.run.smithi005.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.354 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.354 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T12:00:21.356 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T12:00:21.501 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:00:21.528 INFO:teuthology.orchestra.run.smithi037.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.528 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.528 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T12:00:21.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T12:00:21.675 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:00:21.701 INFO:teuthology.orchestra.run.smithi039.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.701 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.701 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T12:00:21.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T12:00:21.853 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T12:00:21.875 INFO:teuthology.orchestra.run.smithi087.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.875 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T12:00:21.875 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T12:00:21.877 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T12:00:21.877 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:21.877 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:00:21.910 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:21.910 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:00:21.943 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:21.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:00:21.975 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:21.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T12:00:22.004 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T12:00:22.004 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:22.005 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:00:22.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:00:22.106 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:22.107 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:00:22.137 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:00:22.209 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:22.209 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:00:22.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:00:22.310 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:22.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T12:00:22.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T12:00:22.412 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T12:00:22.412 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:22.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:00:22.444 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:00:22.513 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:22.513 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:00:22.544 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:00:22.614 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:22.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:00:22.645 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:00:22.715 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:22.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T12:00:22.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T12:00:22.815 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T12:00:22.815 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:22.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:00:22.846 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:00:22.917 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:22.918 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:00:22.948 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:00:23.018 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:23.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:00:23.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:00:23.118 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:23.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T12:00:23.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T12:00:23.217 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T12:00:23.304 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T12:00:23.304 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-09-13T12:00:23.307 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-09-13T12:00:23.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-09-13T12:00:23.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2024-09-13T12:00:23.341 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T12:00:23.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T12:00:23.379 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T12:00:23.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T12:00:23.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T12:00:23.412 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T12:00:23.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T12:00:23.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T12:00:23.452 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T12:00:23.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T12:00:23.494 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T12:00:23.494 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T12:00:23.494 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903463', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 4159, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '682/13280', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFP/DjLr7LjZFoEx3rZfFyWy/KEE8z8Cpl8G75Onpjj/KCHef4+sE3K/CRmkBYIg/xQhuHJRLjZ/m0bIKZwyMZ0=', 'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBayXWhE28aRG7QgefN9CMCdB16Da3RkecGxxr5dV5Ui1QSDDmSRbJ+Noo7DPdf6XjnRrDZmUseXXUP49i8Q9o=', 'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO66PEVIhtIuAaOuMGEaeoD5Jh/5mzNtsSx1/090Wcfk2ugcI5EeHxznnsIVx4Nf54OdPRkozmXohddewD9haMs=', 'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAb2+9Jxq9B/Ij8scRZgoWoNMe3Rcuy8lJ3ynQnma8E5ZWqbIo8jMXmZ8Fm4y/LuTCmfvLezngNp8HRxWvfqJ24='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'cluster': 'ceph'}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 60', 'sudo ceph osd pool set cache target_max_objects 250', 'sudo ceph osd pool set cache min_read_recency_for_promote 2', 'sudo ceph osd pool set cache min_write_recency_for_promote 2']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:43:51', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T12:00:23.495 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T12:00:23.520 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T12:00:23.523 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T12:00:23.525 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T12:00:23.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T12:00:23.577 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T12:00:23.580 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T12:00:23.582 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T12:00:23.609 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:23.609 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:00:23.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:00:23.659 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 12:00:12.094203411 +0000 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:57:35.539816342 +0000 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:57:35.539816342 +0000 2024-09-13T12:00:23.716 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-13 11:57:35.539816342 +0000 2024-09-13T12:00:23.717 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:00:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T12:00:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T12:00:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000105043 s, 4.9 MB/s 2024-09-13T12:00:23.782 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:00:23.840 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 12:00:12.095203390 +0000 2024-09-13T12:00:23.897 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:57:35.875809320 +0000 2024-09-13T12:00:23.898 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:57:35.875809320 +0000 2024-09-13T12:00:23.898 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-13 11:57:35.875809320 +0000 2024-09-13T12:00:23.898 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:00:23.962 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T12:00:23.963 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T12:00:23.963 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00011387 s, 4.5 MB/s 2024-09-13T12:00:23.964 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:00:24.022 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 12:00:12.095203390 +0000 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:57:36.208802362 +0000 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:57:36.208802362 +0000 2024-09-13T12:00:24.078 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-13 11:57:36.208802362 +0000 2024-09-13T12:00:24.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:00:24.142 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T12:00:24.142 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T12:00:24.142 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000102037 s, 5.0 MB/s 2024-09-13T12:00:24.143 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:00:24.200 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-09-13T12:00:24.256 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-09-13 12:00:12.096203369 +0000 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-09-13 11:57:36.533795571 +0000 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-09-13 11:57:36.533795571 +0000 2024-09-13T12:00:24.257 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-09-13 11:57:36.533795571 +0000 2024-09-13T12:00:24.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:00:24.321 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-09-13T12:00:24.321 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-09-13T12:00:24.321 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000101219 s, 5.1 MB/s 2024-09-13T12:00:24.322 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:00:24.379 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:24.380 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:24.380 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:00:24.401 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:00:24.401 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-09-13T12:00:24.462 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:00:24.462 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-13T12:00:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:24.462 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-13 12:00:10.223382117 +0000 2024-09-13T12:00:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-13 11:57:37.029626067 +0000 2024-09-13T12:00:24.463 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-13 11:57:37.029626067 +0000 2024-09-13T12:00:24.463 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-09-13 11:57:37.029626067 +0000 2024-09-13T12:00:24.463 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:00:24.531 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-13T12:00:24.531 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-13T12:00:24.531 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000297743 s, 1.7 MB/s 2024-09-13T12:00:24.533 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:00:24.593 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-09-13T12:00:24.653 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-13 12:00:10.223382117 +0000 2024-09-13T12:00:24.654 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-13 11:57:37.381621615 +0000 2024-09-13T12:00:24.655 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-13 11:57:37.381621615 +0000 2024-09-13T12:00:24.655 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-09-13 11:57:37.381621615 +0000 2024-09-13T12:00:24.655 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:00:24.721 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-13T12:00:24.721 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-13T12:00:24.721 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000208072 s, 2.5 MB/s 2024-09-13T12:00:24.723 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:00:24.782 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-09-13T12:00:24.839 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:00:24.839 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T12:00:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-13 12:00:10.223382117 +0000 2024-09-13T12:00:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-13 11:57:37.709617467 +0000 2024-09-13T12:00:24.840 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-13 11:57:37.709617467 +0000 2024-09-13T12:00:24.840 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-09-13 11:57:37.709617467 +0000 2024-09-13T12:00:24.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:00:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-13T12:00:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-13T12:00:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000201509 s, 2.5 MB/s 2024-09-13T12:00:24.906 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:00:24.964 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-09-13 12:00:10.224382104 +0000 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-09-13 11:57:38.049613167 +0000 2024-09-13T12:00:25.021 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-09-13 11:57:38.049613167 +0000 2024-09-13T12:00:25.022 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-09-13 11:57:38.049613167 +0000 2024-09-13T12:00:25.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:00:25.086 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-09-13T12:00:25.086 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-09-13T12:00:25.087 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.00021086 s, 2.4 MB/s 2024-09-13T12:00:25.088 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:00:25.145 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:25.145 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:25.145 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:00:25.164 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:00:25.165 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 12:00:16.370457983 +0000 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:57:40.871569990 +0000 2024-09-13T12:00:25.223 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:57:40.871569990 +0000 2024-09-13T12:00:25.224 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-13 11:57:40.871569990 +0000 2024-09-13T12:00:25.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:00:25.289 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T12:00:25.290 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T12:00:25.290 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000320806 s, 1.6 MB/s 2024-09-13T12:00:25.291 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:00:25.349 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-13T12:00:25.406 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:00:25.406 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 12:00:16.370457983 +0000 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:57:41.209563635 +0000 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:57:41.209563635 +0000 2024-09-13T12:00:25.407 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-13 11:57:41.209563635 +0000 2024-09-13T12:00:25.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:00:25.472 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T12:00:25.472 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T12:00:25.472 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000280342 s, 1.8 MB/s 2024-09-13T12:00:25.473 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:00:25.531 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 12:00:16.371457964 +0000 2024-09-13T12:00:25.588 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:57:41.539557430 +0000 2024-09-13T12:00:25.589 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:57:41.539557430 +0000 2024-09-13T12:00:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-13 11:57:41.539557430 +0000 2024-09-13T12:00:25.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:00:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T12:00:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T12:00:25.654 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00038471 s, 1.3 MB/s 2024-09-13T12:00:25.656 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:00:25.713 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-13 12:00:16.371457964 +0000 2024-09-13T12:00:25.770 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-13 11:57:41.887550888 +0000 2024-09-13T12:00:25.771 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-13 11:57:41.887550888 +0000 2024-09-13T12:00:25.771 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-09-13 11:57:41.887550888 +0000 2024-09-13T12:00:25.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:00:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-13T12:00:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-13T12:00:25.835 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00030823 s, 1.7 MB/s 2024-09-13T12:00:25.836 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:00:25.894 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:25.894 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:25.894 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T12:00:25.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T12:00:25.912 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-09-13T12:00:25.970 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-13 12:00:17.911577554 +0000 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-13 11:57:30.623899503 +0000 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-13 11:57:30.623899503 +0000 2024-09-13T12:00:25.971 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-13 11:57:30.623899503 +0000 2024-09-13T12:00:25.972 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T12:00:26.039 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-13T12:00:26.039 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-13T12:00:26.039 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000216237 s, 2.4 MB/s 2024-09-13T12:00:26.040 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T12:00:26.098 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-09-13T12:00:26.154 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T12:00:26.154 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-13 12:00:17.912577532 +0000 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-13 11:57:30.951892132 +0000 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-13 11:57:30.951892132 +0000 2024-09-13T12:00:26.155 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-13 11:57:30.951892132 +0000 2024-09-13T12:00:26.155 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T12:00:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-13T12:00:26.220 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-13T12:00:26.221 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000207868 s, 2.5 MB/s 2024-09-13T12:00:26.222 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T12:00:26.279 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-09-13T12:00:26.335 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T12:00:26.335 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-13 12:00:17.912577532 +0000 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-13 11:57:31.273884895 +0000 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-13 11:57:31.273884895 +0000 2024-09-13T12:00:26.336 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-13 11:57:31.273884895 +0000 2024-09-13T12:00:26.336 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T12:00:26.402 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-13T12:00:26.402 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-13T12:00:26.402 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000205517 s, 2.5 MB/s 2024-09-13T12:00:26.404 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T12:00:26.461 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-09-13T12:00:26.516 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T12:00:26.516 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-09-13 12:00:17.913577509 +0000 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-09-13 11:57:31.599877569 +0000 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-09-13 11:57:31.599877569 +0000 2024-09-13T12:00:26.517 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2024-09-13 11:57:31.599877569 +0000 2024-09-13T12:00:26.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T12:00:26.580 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-09-13T12:00:26.580 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-09-13T12:00:26.580 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000167712 s, 3.1 MB/s 2024-09-13T12:00:26.581 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T12:00:26.637 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:26.637 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T12:00:26.637 INFO:tasks.ceph:Generating config... 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T12:00:26.639 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd beacon report interval = 30 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd map cache size = 1 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T12:00:26.640 INFO:tasks.ceph:[osd] osd max scrubs = 5 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2024-09-13T12:00:26.641 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2024-09-13T12:00:26.641 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T12:00:26.641 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-13T12:00:26.641 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-13T12:00:26.641 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] ms type = async 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-13T12:00:26.642 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-13T12:00:26.642 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T12:00:26.643 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T12:00:26.643 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T12:00:26.643 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T12:00:26.643 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-13T12:00:26.643 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T12:00:26.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T12:00:26.697 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T12:00:26.701 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T12:00:26.749 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T12:00:26.776 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5'), ('mon.b', '172.21.15.37'), ('mon.c', '172.21.15.39')] 2024-09-13T12:00:26.777 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5,172.21.15.37,172.21.15.39', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-13T12:00:26.778 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:26.778 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T12:00:26.833 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --add b 172.21.15.37 --add c 172.21.15.39 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:26.923 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:26.923 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:26.923 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-09-13T12:00:26.923257+0000 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-13T12:00:26.923257+0000 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 17 (quincy) 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:1: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.b 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:2: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.c 2024-09-13T12:00:26.924 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T12:00:26.926 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T12:00:26.941 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0694c701-d7eb-494e-ad86-64faa4809c32... 2024-09-13T12:00:26.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T12:00:26.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T12:00:26.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T12:00:26.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-09-13T12:00:27.028 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T12:00:27.029 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-09-13T12:00:27.030 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-09-13T12:00:27.031 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.5,172.21.15.37,172.21.15.39" 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: mon client directed command retry = 5 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: mon election default strategy = 1 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr1 = False 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: ms bind msgr2 = True 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: ms inject socket failures = 2500 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: ms tcp read timeout = 5 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: ms type = async 2024-09-13T12:00:27.032 INFO:teuthology.orchestra.run.smithi005.stdout: osd_async_recovery_min_cost = 1 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_min_size = 2 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_size = 2 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 120 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-09-13T12:00:27.033 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-09-13T12:00:27.034 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug trim objects = true 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-09-13T12:00:27.035 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: debug monc = 20 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bdev async discard = True 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bdev enable discard = True 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore allocator = bitmap 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore write v2 random = True 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore zero block detection = True 2024-09-13T12:00:27.036 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: osd beacon report interval = 30 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: osd blocked scrub grace period = 3600 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify cached snaps = True 2024-09-13T12:00:27.037 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify missing on start = True 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd inject bad map crc probability = 0.1 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd map cache size = 1 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd max backfills = 6 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd max markdown count = 1000 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd max scrubs = 5 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock override recovery settings = True 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue = debug_random 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd op queue cut off = debug_random 2024-09-13T12:00:27.038 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub during recovery = False 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub min interval = 60 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: osd shutdown pgref assert = True 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: osd_max_pg_log_entries = 3000 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: osd_min_pg_log_entries = 3000 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.039 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-09-13T12:00:27.040 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-09-13T12:00:27.041 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon min osdmap epochs = 50 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune interval = 2 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune min = 15 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon osdmap full prune txsize = 2 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: mon scrub interval = 300 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: paxos service trim min = 10 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-09-13T12:00:27.042 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-09-13T12:00:27.043 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T12:00:27.043 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T12:00:27.043 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-09-13T12:00:27.043 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-09-13T12:00:27.043 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-09-13T12:00:27.044 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T12:00:27.045 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-09-13T12:00:27.046 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-09-13T12:00:27.047 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.5,172.21.15.37,172.21.15.39" 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: mon client directed command retry = 5 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: mon election default strategy = 1 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: ms bind msgr1 = False 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: ms bind msgr2 = True 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject socket failures = 2500 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: ms tcp read timeout = 5 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: ms type = async 2024-09-13T12:00:27.048 INFO:teuthology.orchestra.run.smithi039.stdout: osd_async_recovery_min_cost = 1 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_min_size = 2 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_size = 2 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 120 2024-09-13T12:00:27.049 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-09-13T12:00:27.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-09-13T12:00:27.051 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore write v2 random = True 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore zero block detection = True 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 20 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 20 2024-09-13T12:00:27.052 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 10 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd beacon report interval = 30 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd blocked scrub grace period = 3600 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify cached snaps = True 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify missing on start = True 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-09-13T12:00:27.053 INFO:teuthology.orchestra.run.smithi039.stdout: osd inject bad map crc probability = 0.1 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd map cache size = 1 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd max backfills = 6 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd max markdown count = 1000 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd max scrubs = 5 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock override recovery settings = True 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue = debug_random 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd op queue cut off = debug_random 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub during recovery = False 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub min interval = 60 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd shutdown pgref assert = True 2024-09-13T12:00:27.054 INFO:teuthology.orchestra.run.smithi039.stdout: osd_max_pg_log_entries = 3000 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: osd_min_pg_log_entries = 3000 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-09-13T12:00:27.055 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.056 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.057 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: mon min osdmap epochs = 50 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune interval = 2 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune min = 15 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: mon osdmap full prune txsize = 2 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: mon scrub interval = 300 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: paxos service trim min = 10 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-09-13T12:00:27.058 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-09-13T12:00:27.059 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-09-13T12:00:27.060 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-09-13T12:00:27.061 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-09-13T12:00:27.062 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.5,172.21.15.37,172.21.15.39" 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 1 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr1 = False 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr2 = True 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 2500 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: ms tcp read timeout = 5 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: ms type = async 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: osd_async_recovery_min_cost = 1 2024-09-13T12:00:27.063 INFO:teuthology.orchestra.run.smithi037.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_min_size = 2 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_size = 2 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 120 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.064 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-09-13T12:00:27.065 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-09-13T12:00:27.066 INFO:teuthology.orchestra.run.smithi037.stdout: bdev async discard = True 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bdev enable discard = True 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore allocator = bitmap 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore write v2 random = True 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-09-13T12:00:27.067 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd beacon report interval = 30 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd blocked scrub grace period = 3600 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd inject bad map crc probability = 0.1 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd map cache size = 1 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd max backfills = 6 2024-09-13T12:00:27.068 INFO:teuthology.orchestra.run.smithi037.stdout: osd max markdown count = 1000 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd max scrubs = 5 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub during recovery = False 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub min interval = 60 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd shutdown pgref assert = True 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd_max_pg_log_entries = 3000 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: osd_min_pg_log_entries = 3000 2024-09-13T12:00:27.069 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-09-13T12:00:27.070 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T12:00:27.071 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon min osdmap epochs = 50 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune interval = 2 2024-09-13T12:00:27.072 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune min = 15 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune txsize = 2 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: paxos service trim min = 10 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-09-13T12:00:27.073 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2024-09-13T12:00:27.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T12:00:27.075 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2024-09-13T12:00:27.076 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.077 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = "172.21.15.5,172.21.15.37,172.21.15.39" 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon election default strategy = 1 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr1 = False 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: ms bind msgr2 = True 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 2500 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: ms tcp read timeout = 5 2024-09-13T12:00:27.078 INFO:teuthology.orchestra.run.smithi087.stdout: ms type = async 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd_async_recovery_min_cost = 1 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_min_size = 2 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_size = 2 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 120 2024-09-13T12:00:27.079 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2024-09-13T12:00:27.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2024-09-13T12:00:27.081 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: debug monc = 20 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = bitmap 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore write v2 random = True 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore zero block detection = True 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 20 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 20 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-13T12:00:27.082 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 10 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd beacon report interval = 30 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd blocked scrub grace period = 3600 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify cached snaps = True 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify missing on start = True 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd inject bad map crc probability = 0.1 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd map cache size = 1 2024-09-13T12:00:27.083 INFO:teuthology.orchestra.run.smithi087.stdout: osd max backfills = 6 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd max markdown count = 1000 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd max scrubs = 5 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock override recovery settings = True 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue = debug_random 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd op queue cut off = debug_random 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub during recovery = False 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub min interval = 60 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd shutdown pgref assert = True 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd_max_pg_log_entries = 3000 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: osd_min_pg_log_entries = 3000 2024-09-13T12:00:27.084 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2024-09-13T12:00:27.085 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T12:00:27.086 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon min osdmap epochs = 50 2024-09-13T12:00:27.087 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune interval = 2 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune min = 15 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: mon osdmap full prune txsize = 2 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: mon scrub interval = 300 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: paxos service trim min = 10 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T12:00:27.088 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2024-09-13T12:00:27.089 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.b] 2024-09-13T12:00:27.089 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.c] 2024-09-13T12:00:27.091 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T12:00:27.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T12:00:27.186 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T12:00:27.186 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:27.187 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T12:00:27.202 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:27.202 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T12:00:27.258 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:00:27.258 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:27.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T12:00:27.259 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T12:00:27.335 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:00:27.335 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:27.391 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:00:27.392 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:27.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T12:00:27.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T12:00:27.434 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:00:27.434 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:27.492 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:00:27.492 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:27.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T12:00:27.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T12:00:27.535 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:00:27.535 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:27.594 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:00:27.594 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:27.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T12:00:27.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T12:00:27.635 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:00:27.635 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:27.693 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T12:00:27.693 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T12:00:27.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T12:00:27.748 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T12:00:27.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T12:00:27.829 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T12:00:27.832 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T12:00:27.832 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T12:00:27.832 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T12:00:27.876 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T12:00:27.889 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T12:00:27.889 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T12:00:27.889 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T12:00:27.915 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:27.916 INFO:tasks.ceph:role: osd.0 2024-09-13T12:00:27.916 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:00:27.916 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T12:00:27.981 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:27.982 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:28.014 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T12:00:28.017 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T12:00:28.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T12:00:28.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T12:00:28.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T12:00:28.222 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:28.222 INFO:tasks.ceph:role: osd.4 2024-09-13T12:00:28.223 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:00:28.223 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T12:00:28.288 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:28.289 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:28.322 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T12:00:28.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T12:00:28.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T12:00:28.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T12:00:28.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-13T12:00:28.531 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:28.531 INFO:tasks.ceph:role: osd.8 2024-09-13T12:00:28.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:00:28.531 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T12:00:28.596 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:28.596 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:28.597 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:28.629 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T12:00:28.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T12:00:28.631 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-13T12:00:28.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-13T12:00:28.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-13T12:00:28.839 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:28.839 INFO:tasks.ceph:role: osd.12 2024-09-13T12:00:28.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:00:28.840 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:28.905 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:28.941 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-09-13T12:00:28.941 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-09-13T12:00:28.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-13T12:00:29.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-13T12:00:29.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:29.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:29.178+0000 7fc71de87640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T12:00:29.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:29.178+0000 7fc71de87640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T12:00:29.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:29.178+0000 7fc71de87640 -1 bdev(0x5629423eb800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:29.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:29.179+0000 7fc71de87640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T12:00:32.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T12:00:32.386 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:32.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:32.477+0000 7fe7ce20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T12:00:32.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:32.477+0000 7fe7ce20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T12:00:32.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:32.477+0000 7fe7ce20f640 -1 bdev(0x5648fca4b800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:32.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:32.477+0000 7fe7ce20f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T12:00:35.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T12:00:35.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:35.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:35.520+0000 7fe469e13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-13T12:00:35.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:35.521+0000 7fe469e13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-13T12:00:35.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:35.521+0000 7fe469e13640 -1 bdev(0x5583879ff800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:35.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:35.521+0000 7fe469e13640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-13T12:00:38.431 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-13T12:00:38.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:38.554+0000 7f7d41958640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-13T12:00:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:38.555+0000 7f7d41958640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-13T12:00:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:38.555+0000 7f7d41958640 -1 bdev(0x558729e63800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:00:38.555+0000 7f7d41958640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-13T12:00:41.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-13T12:00:41.506 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T12:00:41.537 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:41.537 INFO:tasks.ceph:role: osd.1 2024-09-13T12:00:41.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:00:41.538 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:41.617 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:41.618 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:41.618 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:42.135 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-13T12:00:42.138 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-13T12:00:42.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-13T12:00:42.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T12:00:42.320 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T12:00:42.389 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:42.389 INFO:tasks.ceph:role: osd.5 2024-09-13T12:00:42.389 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:00:42.390 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:42.463 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:42.464 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:42.464 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:42.911 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-13T12:00:42.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-13T12:00:42.913 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T12:00:42.992 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T12:00:43.062 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-13T12:00:43.129 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:43.129 INFO:tasks.ceph:role: osd.9 2024-09-13T12:00:43.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:00:43.129 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T12:00:43.203 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:43.203 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:43.203 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:43.203 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:43.203 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:43.204 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:43.204 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:43.204 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:43.204 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:43.204 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:43.692 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-13T12:00:43.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-13T12:00:43.694 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-13T12:00:43.838 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-13T12:00:43.869 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-13T12:00:43.935 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:43.935 INFO:tasks.ceph:role: osd.13 2024-09-13T12:00:43.935 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:00:43.935 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:44.008 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:44.009 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:44.009 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:44.521 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-09-13T12:00:44.524 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-09-13T12:00:44.524 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-13T12:00:44.645 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-13T12:00:44.681 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:44.812+0000 7f90ea60c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T12:00:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:44.813+0000 7f90ea60c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T12:00:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:44.813+0000 7f90ea60c640 -1 bdev(0x55591c3c7800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:44.813+0000 7f90ea60c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T12:00:47.972 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T12:00:48.004 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:48.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:48.096+0000 7f5376220640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T12:00:48.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:48.096+0000 7f5376220640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T12:00:48.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:48.096+0000 7f5376220640 -1 bdev(0x55a9145a9800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:48.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:48.096+0000 7f5376220640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T12:00:51.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T12:00:51.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:51.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:51.133+0000 7f547e864640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-13T12:00:51.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:51.133+0000 7f547e864640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-13T12:00:51.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:51.133+0000 7f547e864640 -1 bdev(0x55d138673800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:51.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:51.133+0000 7f547e864640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-13T12:00:54.059 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-13T12:00:54.089 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:00:54.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:54.179+0000 7f0f88fd7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-13T12:00:54.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:54.179+0000 7f0f88fd7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-13T12:00:54.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:54.179+0000 7f0f88fd7640 -1 bdev(0x564569c95800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-13T12:00:54.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-09-13T12:00:54.180+0000 7f0f88fd7640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-13T12:00:57.340 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-13T12:00:57.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T12:00:57.405 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:57.405 INFO:tasks.ceph:role: osd.2 2024-09-13T12:00:57.406 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:00:57.406 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:57.504 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:00:58.821 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T12:00:58.824 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T12:00:58.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-13T12:00:58.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T12:00:58.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T12:00:58.995 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T12:00:58.995 INFO:tasks.ceph:role: osd.6 2024-09-13T12:00:58.995 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:00:58.995 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:00:59.088 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:00:59.089 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:00:59.089 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:00:59.089 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:00:59.089 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:00.429 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T12:01:00.432 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T12:01:00.432 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-13T12:01:00.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T12:01:00.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-13T12:01:00.613 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:00.613 INFO:tasks.ceph:role: osd.10 2024-09-13T12:01:00.613 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:01:00.613 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T12:01:00.707 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:00.708 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:00.708 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:00.708 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:00.708 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:00.708 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:00.709 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:00.709 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:00.709 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:00.709 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:02.026 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T12:01:02.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T12:01:02.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-13T12:01:02.066 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-13T12:01:02.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-13T12:01:02.207 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:02.208 INFO:tasks.ceph:role: osd.14 2024-09-13T12:01:02.208 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:01:02.208 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T12:01:02.303 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:02.304 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:03.628 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-13T12:01:03.631 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-13T12:01:03.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-13T12:01:03.670 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-13T12:01:03.742 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:03.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:03.841+0000 7fd4ca329640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T12:01:03.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:03.841+0000 7fd4ca329640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T12:01:03.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:03.841+0000 7fd4ca329640 -1 bdev(0x5593476d1800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:03.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:03.841+0000 7fd4ca329640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T12:01:06.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T12:01:06.811 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:06.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:06.904+0000 7fc53161c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T12:01:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:06.905+0000 7fc53161c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T12:01:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:06.905+0000 7fc53161c640 -1 bdev(0x559d5b8d3800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:06.905+0000 7fc53161c640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T12:01:09.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T12:01:09.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:09.961+0000 7f10116a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-13T12:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:09.962+0000 7f10116a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-13T12:01:09.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:09.962+0000 7f10116a5640 -1 bdev(0x55c8556d1800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:09.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:09.962+0000 7f10116a5640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-13T12:01:12.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-13T12:01:12.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:13.000+0000 7fa4c27dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-13T12:01:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:13.001+0000 7fa4c27dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-13T12:01:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:13.001+0000 7fa4c27dd640 -1 bdev(0x55919f4f7800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:13.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-13T12:01:13.001+0000 7fa4c27dd640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-13T12:01:15.941 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-13T12:01:15.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T12:01:16.004 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:16.005 INFO:tasks.ceph:role: osd.3 2024-09-13T12:01:16.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:01:16.005 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T12:01:16.081 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:16.081 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:16.081 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:16.081 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:16.081 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:16.082 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:16.082 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:16.082 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:16.082 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:16.082 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:16.602 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-13T12:01:16.604 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-13T12:01:16.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T12:01:16.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T12:01:16.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T12:01:16.859 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:16.859 INFO:tasks.ceph:role: osd.7 2024-09-13T12:01:16.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:01:16.859 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:16.933 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:17.388 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-13T12:01:17.390 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-13T12:01:17.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-13T12:01:17.488 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T12:01:17.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-13T12:01:17.587 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:17.587 INFO:tasks.ceph:role: osd.11 2024-09-13T12:01:17.587 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:01:17.587 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T12:01:17.660 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:17.661 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:18.151 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-13T12:01:18.154 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-13T12:01:18.154 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-13T12:01:18.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-13T12:01:18.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-13T12:01:18.401 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T12:01:18.401 INFO:tasks.ceph:role: osd.15 2024-09-13T12:01:18.401 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:01:18.401 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T12:01:18.476 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T12:01:18.477 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T12:01:18.999 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2024-09-13T12:01:19.002 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2024-09-13T12:01:19.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-13T12:01:19.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-13T12:01:19.176 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:19.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:19.274+0000 7fe5fcab2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T12:01:19.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:19.274+0000 7fe5fcab2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T12:01:19.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:19.274+0000 7fe5fcab2640 -1 bdev(0x55cdbbe89800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:19.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:19.274+0000 7fe5fcab2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T12:01:22.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T12:01:22.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:22.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:22.305+0000 7f4322e2f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T12:01:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:22.305+0000 7f4322e2f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T12:01:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:22.305+0000 7f4322e2f640 -1 bdev(0x563f97fb7800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:22.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:22.305+0000 7f4322e2f640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T12:01:25.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T12:01:25.246 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:25.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:25.335+0000 7f20cdf32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-13T12:01:25.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:25.335+0000 7f20cdf32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-13T12:01:25.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:25.335+0000 7f20cdf32640 -1 bdev(0x56345b8b3800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:25.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:25.335+0000 7f20cdf32640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-13T12:01:28.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-13T12:01:28.521 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:28.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:28.610+0000 7f5007418640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-13T12:01:28.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:28.611+0000 7f5007418640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-13T12:01:28.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:28.611+0000 7f5007418640 -1 bdev(0x55cf288a3800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-13T12:01:28.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:01:28.611+0000 7f5007418640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-13T12:01:32.022 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-13T12:01:32.051 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T12:01:32.052 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:32.052 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T12:01:32.087 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:32.087 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T12:01:32.157 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:32.157 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T12:01:32.225 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:32.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-13T12:01:32.294 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:32.294 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-13T12:01:32.362 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:32.362 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T12:01:32.395 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:32.395 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T12:01:32.466 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:32.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-13T12:01:32.537 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:32.537 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-13T12:01:32.608 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:32.609 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T12:01:32.640 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:32.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T12:01:32.710 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:32.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-13T12:01:32.779 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:32.779 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-13T12:01:32.847 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:32.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T12:01:32.878 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:32.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T12:01:32.946 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:32.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T12:01:33.014 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:33.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-13T12:01:33.081 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:33.081 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-13T12:01:33.148 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:33.149 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T12:01:33.205 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T12:01:33.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T12:01:33.208 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T12:01:33.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.y] 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbKeRm9aWOLBAAVhxybdizDXYJviBCH0UsMw== 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBdKeRm3wrBChAA7nkuG9t4GDc8LAUPTLoUBg== 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBgKeRmvhSNHBAAmotGL6F6nzN3AvUE85lB4g== 2024-09-13T12:01:33.235 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.8] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBjKeRmSZclHxAAd+BWtfdTGrYqz4RUwlpMzw== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.12] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBmKeRmz0geIRAAqxNOHRcffLqVqdZGU8u04g== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBsKeRm5eOAMBAAkeIjrjlskDKdkQswk26NzQ== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBwKeRmLi/OBRAAd2APi6zGOL3Qj+i/olD/TA== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.9] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBzKeRmdcsACBAAvCcpeKqO2OjFXhHCR8IMMQ== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.13] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQB2KeRmTkHCChAAtFmKubBET4yOAN77UeywLw== 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-09-13T12:01:33.236 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQB/KeRmzOM7MhAAkBBuQvn7tCmOCYeDP47VpQ== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCCKeRm+1IGNhAA5wuSRn5SDnr0a9v4PXdU1g== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.10] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCFKeRmi0RnORAAiX9NGh8rXKQrfIoWi2a99g== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.14] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCJKeRmROYjABAAd7Pg59tTqPA684pZZbbIgA== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbKeRmQbtoMRAAKOUTxIMxArE85BO93ER5oQ== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCPKeRmdxloEBAAEID2lT5kyNrNrJHyDFoeCg== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCSKeRmV+JDEhAAmzpjoHdF9/B2TK7RmwXsog== 2024-09-13T12:01:33.237 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.11] 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCVKeRmex4LFBAAFFKfT002okkf+/2TY6ir3w== 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.15] 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQCYKeRmgySFJBAA5wDGsu4Pk33Rm7V6nbeGXg== 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQBbKeRm2/E6NBAA/pK0fXaLrR2cxfkgH58dag== 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBbKeRm9aWOLBAAVhxybdizDXYJviBCH0UsMw== 2024-09-13T12:01:33.238 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBdKeRm3wrBChAA7nkuG9t4GDc8LAUPTLoUBg== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBgKeRmvhSNHBAAmotGL6F6nzN3AvUE85lB4g== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.8] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBjKeRmSZclHxAAd+BWtfdTGrYqz4RUwlpMzw== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.12] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBmKeRmz0geIRAAqxNOHRcffLqVqdZGU8u04g== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBsKeRm5eOAMBAAkeIjrjlskDKdkQswk26NzQ== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBwKeRmLi/OBRAAd2APi6zGOL3Qj+i/olD/TA== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.9] 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBzKeRmdcsACBAAvCcpeKqO2OjFXhHCR8IMMQ== 2024-09-13T12:01:33.239 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.13] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQB2KeRmTkHCChAAtFmKubBET4yOAN77UeywLw== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQB/KeRmzOM7MhAAkBBuQvn7tCmOCYeDP47VpQ== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCCKeRm+1IGNhAA5wuSRn5SDnr0a9v4PXdU1g== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.10] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCFKeRmi0RnORAAiX9NGh8rXKQrfIoWi2a99g== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.14] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCJKeRmROYjABAAd7Pg59tTqPA684pZZbbIgA== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBbKeRmQbtoMRAAKOUTxIMxArE85BO93ER5oQ== 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-09-13T12:01:33.240 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCPKeRmdxloEBAAEID2lT5kyNrNrJHyDFoeCg== 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCSKeRmV+JDEhAAmzpjoHdF9/B2TK7RmwXsog== 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.11] 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCVKeRmex4LFBAAFFKfT002okkf+/2TY6ir3w== 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.15] 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQCYKeRmgySFJBAA5wDGsu4Pk33Rm7V6nbeGXg== 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-09-13T12:01:33.241 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBbKeRm2/E6NBAA/pK0fXaLrR2cxfkgH58dag== 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.y] 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBbKeRm9aWOLBAAVhxybdizDXYJviBCH0UsMw== 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBdKeRm3wrBChAA7nkuG9t4GDc8LAUPTLoUBg== 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBgKeRmvhSNHBAAmotGL6F6nzN3AvUE85lB4g== 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.8] 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBjKeRmSZclHxAAd+BWtfdTGrYqz4RUwlpMzw== 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.12] 2024-09-13T12:01:33.243 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBmKeRmz0geIRAAqxNOHRcffLqVqdZGU8u04g== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBsKeRm5eOAMBAAkeIjrjlskDKdkQswk26NzQ== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBwKeRmLi/OBRAAd2APi6zGOL3Qj+i/olD/TA== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.9] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBzKeRmdcsACBAAvCcpeKqO2OjFXhHCR8IMMQ== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.13] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB2KeRmTkHCChAAtFmKubBET4yOAN77UeywLw== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB/KeRmzOM7MhAAkBBuQvn7tCmOCYeDP47VpQ== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.6] 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCCKeRm+1IGNhAA5wuSRn5SDnr0a9v4PXdU1g== 2024-09-13T12:01:33.244 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.10] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCFKeRmi0RnORAAiX9NGh8rXKQrfIoWi2a99g== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.14] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCJKeRmROYjABAAd7Pg59tTqPA684pZZbbIgA== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBbKeRmQbtoMRAAKOUTxIMxArE85BO93ER5oQ== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCPKeRmdxloEBAAEID2lT5kyNrNrJHyDFoeCg== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.7] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCSKeRmV+JDEhAAmzpjoHdF9/B2TK7RmwXsog== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.11] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCVKeRmex4LFBAAFFKfT002okkf+/2TY6ir3w== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.15] 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCYKeRmgySFJBAA5wDGsu4Pk33Rm7V6nbeGXg== 2024-09-13T12:01:33.245 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-13T12:01:33.246 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBbKeRm2/E6NBAA/pK0fXaLrR2cxfkgH58dag== 2024-09-13T12:01:33.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:33.285 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:33.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:33.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.348 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.351 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.409 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.411 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.464 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.469 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.521 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.523 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.526 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.581 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.634 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.637 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.639 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.692 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.694 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.750 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.752 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.811 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.868 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.870 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.922 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.925 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.980 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.982 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:33.985 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:34.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:34.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T12:01:34.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.097 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.151 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.154 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.208 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.211 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.216 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.271 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T12:01:34.325 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T12:01:34.327 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T12:01:34.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T12:01:34.385 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T12:01:34.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T12:01:34.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T12:01:34.508 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T12:01:34.534 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T12:01:34.560 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T12:01:34.657 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T12:01:34.684 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T12:01:34.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T12:01:34.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T12:01:34.842 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:34.845 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:34.847 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T12:01:34.898 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T12:01:34.899 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T12:01:34.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T12:01:34.901 INFO:tasks.ceph.mon.a:Started 2024-09-13T12:01:34.901 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T12:01:34.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T12:01:34.904 INFO:tasks.ceph.mon.b:Started 2024-09-13T12:01:34.904 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T12:01:34.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T12:01:34.941 INFO:tasks.ceph.mon.c:Started 2024-09-13T12:01:34.941 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T12:01:34.941 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T12:01:34.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T12:01:34.944 INFO:tasks.ceph.mgr.y:Started 2024-09-13T12:01:34.944 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T12:01:34.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T12:01:34.946 INFO:tasks.ceph.mgr.x:Started 2024-09-13T12:01:34.947 DEBUG:tasks.ceph:set 0 configs 2024-09-13T12:01:34.947 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T12:01:34.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T12:01:45.368 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-09-13T12:01:45.380 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T12:01:45.380 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T12:01:45.381 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:45.381 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T12:01:45.407 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:45.407 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T12:01:45.472 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:45.472 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-13T12:01:45.538 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-09-13T12:01:45.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-13T12:01:45.604 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:45.605 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T12:01:45.637 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:45.637 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T12:01:45.706 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:45.706 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-13T12:01:45.775 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-09-13T12:01:45.775 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-13T12:01:45.843 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:45.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T12:01:45.875 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:45.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T12:01:45.945 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:45.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-13T12:01:46.014 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-13T12:01:46.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-13T12:01:46.081 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:46.081 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T12:01:46.105 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:46.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T12:01:46.170 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:46.170 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-13T12:01:46.235 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-09-13T12:01:46.235 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-13T12:01:46.300 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 17c551c2-db05-4b58-af49-885510ae31ae 0 2024-09-13T12:01:46.520 INFO:teuthology.orchestra.run.smithi087.stdout:0 2024-09-13T12:01:46.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new a19df369-47ed-4f8c-a422-f009d11c2360 1 2024-09-13T12:01:46.709 INFO:teuthology.orchestra.run.smithi087.stdout:1 2024-09-13T12:01:46.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e 2 2024-09-13T12:01:46.899 INFO:teuthology.orchestra.run.smithi087.stdout:2 2024-09-13T12:01:46.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 6b725dee-2e37-4fd1-a123-4d38d0e6979e 3 2024-09-13T12:01:47.095 INFO:teuthology.orchestra.run.smithi087.stdout:3 2024-09-13T12:01:47.107 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 1b99fff1-9eb6-4883-bc7d-7422a005750d 4 2024-09-13T12:01:47.286 INFO:teuthology.orchestra.run.smithi087.stdout:4 2024-09-13T12:01:47.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 148d447f-4841-4fe1-8051-5f3f0094e5d1 5 2024-09-13T12:01:47.476 INFO:teuthology.orchestra.run.smithi087.stdout:5 2024-09-13T12:01:47.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new ac4ec297-ccbb-4bbe-94b3-4859b1a7851a 6 2024-09-13T12:01:47.671 INFO:teuthology.orchestra.run.smithi087.stdout:6 2024-09-13T12:01:47.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new f8b5938e-5b74-4ef8-bb86-508b56a05296 7 2024-09-13T12:01:47.859 INFO:teuthology.orchestra.run.smithi087.stdout:7 2024-09-13T12:01:47.871 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 407a9a27-7549-489e-8a12-b0951ac5dc26 8 2024-09-13T12:01:48.050 INFO:teuthology.orchestra.run.smithi087.stdout:8 2024-09-13T12:01:48.061 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 4f199848-02c4-43ac-8344-36cd6dac6034 9 2024-09-13T12:01:48.245 INFO:teuthology.orchestra.run.smithi087.stdout:9 2024-09-13T12:01:48.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b 10 2024-09-13T12:01:48.433 INFO:teuthology.orchestra.run.smithi087.stdout:10 2024-09-13T12:01:48.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 52c5aaa9-47d2-41da-a8f2-bb1c69f90259 11 2024-09-13T12:01:48.622 INFO:teuthology.orchestra.run.smithi087.stdout:11 2024-09-13T12:01:48.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 96321c3f-1589-4749-ba88-b48cdd72140d 12 2024-09-13T12:01:48.814 INFO:teuthology.orchestra.run.smithi087.stdout:12 2024-09-13T12:01:48.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 4942dad7-9e60-41a8-8098-12de4684b4f1 13 2024-09-13T12:01:49.002 INFO:teuthology.orchestra.run.smithi087.stdout:13 2024-09-13T12:01:49.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new 1fa4c21c-7e24-49bb-9065-f8a189b9676b 14 2024-09-13T12:01:49.209 INFO:teuthology.orchestra.run.smithi087.stdout:14 2024-09-13T12:01:49.295 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster ceph osd new aba98af9-0ce7-4292-97fc-f15df5e45ddb 15 2024-09-13T12:01:49.476 INFO:teuthology.orchestra.run.smithi087.stdout:15 2024-09-13T12:01:49.486 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T12:01:49.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T12:01:49.489 INFO:tasks.ceph.osd.0:Started 2024-09-13T12:01:49.489 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T12:01:49.489 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T12:01:49.491 INFO:tasks.ceph.osd.4:Started 2024-09-13T12:01:49.492 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-13T12:01:49.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-13T12:01:49.494 INFO:tasks.ceph.osd.8:Started 2024-09-13T12:01:49.494 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-13T12:01:49.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-13T12:01:49.497 INFO:tasks.ceph.osd.12:Started 2024-09-13T12:01:49.497 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T12:01:49.497 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T12:01:49.500 INFO:tasks.ceph.osd.1:Started 2024-09-13T12:01:49.500 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T12:01:49.500 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T12:01:49.502 INFO:tasks.ceph.osd.5:Started 2024-09-13T12:01:49.502 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-13T12:01:49.503 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-13T12:01:49.505 INFO:tasks.ceph.osd.9:Started 2024-09-13T12:01:49.505 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-13T12:01:49.505 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-13T12:01:49.507 INFO:tasks.ceph.osd.13:Started 2024-09-13T12:01:49.507 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T12:01:49.508 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T12:01:49.510 INFO:tasks.ceph.osd.2:Started 2024-09-13T12:01:49.510 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T12:01:49.510 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T12:01:49.513 INFO:tasks.ceph.osd.6:Started 2024-09-13T12:01:49.513 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-13T12:01:49.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-13T12:01:49.515 INFO:tasks.ceph.osd.10:Started 2024-09-13T12:01:49.516 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-13T12:01:49.516 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-13T12:01:49.518 INFO:tasks.ceph.osd.14:Started 2024-09-13T12:01:49.518 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T12:01:49.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T12:01:49.521 INFO:tasks.ceph.osd.3:Started 2024-09-13T12:01:49.521 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T12:01:49.521 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T12:01:49.523 INFO:tasks.ceph.osd.7:Started 2024-09-13T12:01:49.523 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-13T12:01:49.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-13T12:01:49.526 INFO:tasks.ceph.osd.11:Started 2024-09-13T12:01:49.526 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-13T12:01:49.526 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-13T12:01:49.528 INFO:tasks.ceph.osd.15:Started 2024-09-13T12:01:49.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:01:49.727 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:01:49.727 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":18,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:01:49.473230+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:01:49.737 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T12:01:49.738 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T12:01:50.470 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:01:50.469+0000 7f375da20640 -1 Falling back to public interface 2024-09-13T12:01:50.471 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:01:50.469+0000 7f369482c640 -1 Falling back to public interface 2024-09-13T12:01:50.482 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:50.481+0000 7f4dc1954640 -1 Falling back to public interface 2024-09-13T12:01:50.565 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-09-13T12:01:50.564+0000 7fb13d731640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T12:01:50.565 INFO:tasks.ceph.mgr.x.smithi087.stderr:2024-09-13T12:01:50.564+0000 7fb13d731640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T12:01:50.655 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:01:50.654+0000 7fa067dc3640 -1 Falling back to public interface 2024-09-13T12:01:50.698 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:01:50.696+0000 7f5b98419640 -1 Falling back to public interface 2024-09-13T12:01:50.704 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:01:50.702+0000 7f32a2351640 -1 Falling back to public interface 2024-09-13T12:01:50.718 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:01:50.717+0000 7fc777b89640 -1 Falling back to public interface 2024-09-13T12:01:50.736 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:01:50.735+0000 7ff4f902c640 -1 Falling back to public interface 2024-09-13T12:01:50.945 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:01:50.943+0000 7f1add80b640 -1 Falling back to public interface 2024-09-13T12:01:50.954 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:01:50.953+0000 7f8ca27c5640 -1 Falling back to public interface 2024-09-13T12:01:50.977 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:01:50.975+0000 7f3239620640 -1 Falling back to public interface 2024-09-13T12:01:51.005 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:01:51.004+0000 7fbe4cc11640 -1 Falling back to public interface 2024-09-13T12:01:51.009 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:01:51.007+0000 7f933fc0f640 -1 Falling back to public interface 2024-09-13T12:01:51.226 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:51.224+0000 7f83e49bb640 -1 Falling back to public interface 2024-09-13T12:01:51.251 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:01:51.249+0000 7f4dd8eb4640 -1 Falling back to public interface 2024-09-13T12:01:51.347 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T12:01:51.478 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:51.477+0000 7f6412013640 -1 Falling back to public interface 2024-09-13T12:01:51.653 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T12:01:51.654 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":18,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:01:49.473230+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:01:51.664 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-13T12:01:54.051 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:01:54.050+0000 7f375da20640 -1 osd.12 0 log_to_monitors true 2024-09-13T12:01:54.195 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:01:54.194+0000 7f5b98419640 -1 osd.8 0 log_to_monitors true 2024-09-13T12:01:54.282 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:01:54.281+0000 7ff4f902c640 -1 osd.9 0 log_to_monitors true 2024-09-13T12:01:54.306 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:54.306+0000 7f4dc1954640 -1 osd.0 0 log_to_monitors true 2024-09-13T12:01:54.495 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:01:54.494+0000 7fc777b89640 -1 osd.4 0 log_to_monitors true 2024-09-13T12:01:54.501 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:01:54.500+0000 7f8ca27c5640 -1 osd.5 0 log_to_monitors true 2024-09-13T12:01:54.654 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:01:54.653+0000 7fa067dc3640 -1 osd.7 0 log_to_monitors true 2024-09-13T12:01:54.734 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:01:54.733+0000 7f32a2351640 -1 osd.13 0 log_to_monitors true 2024-09-13T12:01:54.760 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:01:54.758+0000 7f3239620640 -1 osd.10 0 log_to_monitors true 2024-09-13T12:01:54.770 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:01:54.769+0000 7f369482c640 -1 osd.11 0 log_to_monitors true 2024-09-13T12:01:54.771 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:01:54.770+0000 7f1add80b640 -1 osd.1 0 log_to_monitors true 2024-09-13T12:01:54.839 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:01:54.837+0000 7f933fc0f640 -1 osd.6 0 log_to_monitors true 2024-09-13T12:01:55.089 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:01:55.084+0000 7fbe4cc11640 -1 osd.2 0 log_to_monitors true 2024-09-13T12:01:55.118 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:01:55.117+0000 7f37599af640 -1 osd.12 0 waiting for initial osdmap 2024-09-13T12:01:55.119 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:01:55.118+0000 7f37513c9640 -1 osd.12 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:55.124 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:01:55.124+0000 7f37557d6640 -1 osd.12 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:55.482 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:55.480+0000 7f6412013640 -1 osd.14 0 log_to_monitors true 2024-09-13T12:01:55.491 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:55.489+0000 7f83e49bb640 -1 osd.3 0 log_to_monitors true 2024-09-13T12:01:55.797 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:01:55.796+0000 7f4dd8eb4640 -1 osd.15 0 log_to_monitors true 2024-09-13T12:01:56.118 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:56.117+0000 7f4dbd8e3640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T12:01:56.119 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:56.118+0000 7f4db52fd640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:56.151 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:56.151+0000 7f4db970a640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:56.162 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:01:56.160+0000 7f3234f11640 -1 osd.10 0 waiting for initial osdmap 2024-09-13T12:01:56.163 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:01:56.162+0000 7f322c92b640 -1 osd.10 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:56.396 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:01:56.394+0000 7f3230d38640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:56.733 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:01:56.732+0000 7fc773b18640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T12:01:56.734 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:01:56.733+0000 7fc76752a640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:56.738 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:01:56.737+0000 7fc76b937640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.047 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:01:57.046+0000 7f5b93d26640 -1 osd.8 0 waiting for initial osdmap 2024-09-13T12:01:57.048 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:01:57.047+0000 7f5b87739640 -1 osd.8 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.052 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:01:57.051+0000 7f5b8bb45640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.192 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:01:57.191+0000 7fa063d52640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T12:01:57.193 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:01:57.191+0000 7f3690fcf640 -1 osd.11 0 waiting for initial osdmap 2024-09-13T12:01:57.193 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:01:57.192+0000 7fa057764640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.193 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:01:57.192+0000 7f36841cd640 -1 osd.11 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.205 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:01:57.204+0000 7f36885da640 -1 osd.11 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.206 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:01:57.205+0000 7fa05bb71640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.273 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:01:57.272+0000 7f4dd4e43640 -1 osd.15 0 waiting for initial osdmap 2024-09-13T12:01:57.278 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:01:57.277+0000 7f4dd0c6a640 -1 osd.15 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.326 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:57.325+0000 7f83e094a640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T12:01:57.341 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:01:57.340+0000 7fbe48da6640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T12:01:57.342 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:01:57.341+0000 7fbe3bfa4640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.345 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:57.344+0000 7f83d8364640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.366 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:01:57.365+0000 7fbe403b1640 -1 osd.2 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.373 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:01:57.371+0000 7f933b486640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T12:01:57.373 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:01:57.372+0000 7f932ee98640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.396 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:01:57.394+0000 7f93332a5640 -1 osd.6 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.398 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:57.397+0000 7f83d8364640 -1 osd.3 1 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.449 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:01:57.448+0000 7f83dc771640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.834 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:57.832+0000 7f640d906640 -1 osd.14 0 waiting for initial osdmap 2024-09-13T12:01:57.834 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:57.833+0000 7f6401318640 -1 osd.14 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.839 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:57.838+0000 7f6405725640 -1 osd.14 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:57.887 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:01:57.885+0000 7f8c9e754640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T12:01:57.889 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:01:57.887+0000 7f8c92166640 -1 osd.5 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:57.914 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:01:57.912+0000 7f8c96573640 -1 osd.5 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:58.010 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:01:58.008+0000 7ff4f4585640 -1 osd.9 0 waiting for initial osdmap 2024-09-13T12:01:58.010 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:01:58.008+0000 7ff4e7f97640 -1 osd.9 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:58.017 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:01:58.015+0000 7ff4ec3a4640 -1 osd.9 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:58.019 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:01:58.017+0000 7f1ad9231640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T12:01:58.019 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:01:58.017+0000 7f1accc43640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:58.026 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:01:58.024+0000 7f1ad1050640 -1 osd.1 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:58.038 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:01:58.037+0000 7f329eaf4640 -1 osd.13 0 waiting for initial osdmap 2024-09-13T12:01:58.038 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:01:58.037+0000 7f3291cf2640 -1 osd.13 0 handle_osd_map injecting map crc failure 2024-09-13T12:01:58.048 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:01:58.046+0000 7f32960ff640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T12:01:58.281 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:58.279+0000 7f6401318640 -1 osd.14 22 handle_osd_map injecting map crc failure 2024-09-13T12:01:59.273 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T12:01:59.437 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:01:59.435+0000 7f63ff314640 -1 osd.14 23 handle_osd_map injecting map crc failure 2024-09-13T12:01:59.462 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:01:59.460+0000 7f4db32f9640 -1 osd.0 22 handle_osd_map injecting map crc failure 2024-09-13T12:01:59.542 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:01:59.541+0000 7f4dca859640 -1 osd.15 23 handle_osd_map injecting map crc failure 2024-09-13T12:01:59.581 INFO:teuthology.misc.health.smithi005.stdout: 2024-09-13T12:01:59.582 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":24,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:01:59.282994+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:01:59.593 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-13T12:01:59.593 INFO:tasks.ceph:Creating RBD pool 2024-09-13T12:01:59.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T12:02:00.300 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'rbd' created 2024-09-13T12:02:00.314 DEBUG:teuthology.orchestra.run.smithi005:> rbd --cluster ceph pool init rbd 2024-09-13T12:02:00.598 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:00.596+0000 7f322a927640 -1 osd.10 23 handle_osd_map injecting map crc failure 2024-09-13T12:02:03.321 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:03.320+0000 7f3291cf2640 -1 osd.13 27 handle_osd_map injecting map crc failure 2024-09-13T12:02:03.341 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T12:02:03.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T12:02:03.342 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T12:02:03.354 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:03.352+0000 7fc765526640 -1 osd.4 26 handle_osd_map injecting map crc failure 2024-09-13T12:02:03.658 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:03.671 INFO:teuthology.orchestra.run.smithi005.stdout:[{"version":12,"timestamp":"2024-09-13T12:01:58.039002+0000","name":"","changes":[{"name":"osd.13/osd_mclock_max_capacity_iops_ssd","new_value":"7074.212722"}]}] 2024-09-13T12:02:03.671 INFO:tasks.ceph_manager:config epoch is 12 2024-09-13T12:02:03.671 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T12:02:03.671 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T12:02:03.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T12:02:04.063 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:04.073 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":153259804}]},"active_addr":"172.21.15.87:6800/153259804","active_change":"2024-09-13T12:01:49.551839+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":1869228282}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2168979810}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3656688875}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3093648383}]}]} 2024-09-13T12:02:04.079 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T12:02:04.079 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T12:02:04.080 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:02:04.381 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:04.381 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":28,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:02:03.318484+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T12:01:59.896737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:02:04.394 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T12:02:04.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:02:04.697 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:04.697 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":28,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:02:03.318484+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T12:01:59.896737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:02:04.710 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T12:02:04.711 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T12:02:04.711 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T12:02:04.711 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T12:02:04.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T12:02:04.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T12:02:04.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T12:02:04.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T12:02:04.713 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T12:02:04.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T12:02:04.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T12:02:04.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T12:02:04.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T12:02:04.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T12:02:04.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T12:02:04.716 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T12:02:05.112 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:05.113 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T12:02:05.153 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-13T12:02:05.153 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T12:02:05.182 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:05.183 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:05.183 INFO:teuthology.orchestra.run.smithi005.stdout:98784247811 2024-09-13T12:02:05.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T12:02:05.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T12:02:05.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T12:02:05.208 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-13T12:02:05.208 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T12:02:05.221 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:05.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T12:02:05.222 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-13T12:02:05.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T12:02:05.225 INFO:teuthology.orchestra.run.smithi005.stdout:90194313220 2024-09-13T12:02:05.226 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T12:02:05.233 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:05.234 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:05.234 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T12:02:05.235 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T12:02:05.242 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:05.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T12:02:05.262 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:05.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T12:02:05.266 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:05.266 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T12:02:05.291 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:05.291 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T12:02:05.349 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-13T12:02:05.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T12:02:06.125 INFO:teuthology.orchestra.run.smithi005.stdout:98784247809 2024-09-13T12:02:06.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247809 for osd.15 2024-09-13T12:02:06.207 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:06.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247812 for osd.2 2024-09-13T12:02:06.254 INFO:teuthology.orchestra.run.smithi005.stdout:98784247811 2024-09-13T12:02:06.258 INFO:teuthology.orchestra.run.smithi005.stdout:98784247810 2024-09-13T12:02:06.273 INFO:teuthology.orchestra.run.smithi005.stdout:98784247811 2024-09-13T12:02:06.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.9 2024-09-13T12:02:06.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247810 for osd.5 2024-09-13T12:02:06.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.7 2024-09-13T12:02:06.411 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-13T12:02:06.424 INFO:teuthology.orchestra.run.smithi005.stdout:90194313220 2024-09-13T12:02:06.424 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:06.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-09-13T12:02:06.432 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.12 2024-09-13T12:02:06.442 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.446 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:06.450 INFO:teuthology.orchestra.run.smithi005.stdout:94489280516 2024-09-13T12:02:06.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.1 2024-09-13T12:02:06.467 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.469 INFO:teuthology.orchestra.run.smithi005.stdout:94489280517 2024-09-13T12:02:06.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.11 2024-09-13T12:02:06.472 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.477 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-09-13T12:02:06.477 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.483 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.8 2024-09-13T12:02:06.483 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.488 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:06.497 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:06.501 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:06.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.3 2024-09-13T12:02:06.507 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.14 2024-09-13T12:02:06.511 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.6 2024-09-13T12:02:06.516 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.525 INFO:teuthology.orchestra.run.smithi005.stdout:94489280515 2024-09-13T12:02:06.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-13T12:02:06.543 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:06.562 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:06.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.13 2024-09-13T12:02:06.575 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T12:02:07.255 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T12:02:07.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T12:02:07.289 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T12:02:07.310 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T12:02:07.510 INFO:teuthology.orchestra.run.smithi005.stdout:98784247811 2024-09-13T12:02:07.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.15 2024-09-13T12:02:07.525 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.594 INFO:teuthology.orchestra.run.smithi005.stdout:98784247814 2024-09-13T12:02:07.612 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247814 got 98784247814 for osd.2 2024-09-13T12:02:07.612 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.682 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:07.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.7 2024-09-13T12:02:07.695 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.695 INFO:teuthology.orchestra.run.smithi005.stdout:98784247812 2024-09-13T12:02:07.701 INFO:teuthology.orchestra.run.smithi005.stdout:98784247813 2024-09-13T12:02:07.710 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.5 2024-09-13T12:02:07.710 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.9 2024-09-13T12:02:07.715 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:07.715 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T12:02:07.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:08.868 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:08.868 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:08.883 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T12:02:07.562434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":449664,"kb_used_data":4780,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509499776,"statfs":{"total":1546188226560,"available":1545727770624,"internally_reserved":0,"allocated":4894720,"data_stored":2645441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.325780+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:03.325780+0000","last_peered":"2024-09-13T12:02:03.325780+0000","last_clean":"2024-09-13T12:02:03.325780+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:03.325780+0000","last_undegraded":"2024-09-13T12:02:03.325780+0000","last_fullsized":"2024-09-13T12:02:03.325780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.322894+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:03.322894+0000","last_peered":"2024-09-13T12:02:03.322894+0000","last_clean":"2024-09-13T12:02:03.322894+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:03.322894+0000","last_undegraded":"2024-09-13T12:02:03.322894+0000","last_fullsized":"2024-09-13T12:02:03.322894+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.358118+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:03.358118+0000","last_peered":"2024-09-13T12:02:03.358118+0000","last_clean":"2024-09-13T12:02:03.358118+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:03.358118+0000","last_undegraded":"2024-09-13T12:02:03.358118+0000","last_fullsized":"2024-09-13T12:02:03.358118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357832+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:03.357832+0000","last_peered":"2024-09-13T12:02:03.357832+0000","last_clean":"2024-09-13T12:02:03.357832+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:03.357832+0000","last_undegraded":"2024-09-13T12:02:03.357832+0000","last_fullsized":"2024-09-13T12:02:03.357832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.328998+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:03.328998+0000","last_peered":"2024-09-13T12:02:03.328998+0000","last_clean":"2024-09-13T12:02:03.328998+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:03.328998+0000","last_undegraded":"2024-09-13T12:02:03.328998+0000","last_fullsized":"2024-09-13T12:02:03.328998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357586+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:03.357586+0000","last_peered":"2024-09-13T12:02:03.357586+0000","last_clean":"2024-09-13T12:02:03.357586+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:03.357586+0000","last_undegraded":"2024-09-13T12:02:03.357586+0000","last_fullsized":"2024-09-13T12:02:03.357586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:04.493017+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:04.493017+0000","last_peered":"2024-09-13T12:02:04.493017+0000","last_clean":"2024-09-13T12:02:04.493017+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:04.493017+0000","last_undegraded":"2024-09-13T12:02:04.493017+0000","last_fullsized":"2024-09-13T12:02:04.493017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357455+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:03.357455+0000","last_peered":"2024-09-13T12:02:03.357455+0000","last_clean":"2024-09-13T12:02:03.357455+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:03.357455+0000","last_undegraded":"2024-09-13T12:02:03.357455+0000","last_fullsized":"2024-09-13T12:02:03.357455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.544429+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:03.544429+0000","last_peered":"2024-09-13T12:02:03.544429+0000","last_clean":"2024-09-13T12:02:03.544429+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:03.544429+0000","last_undegraded":"2024-09-13T12:02:03.544429+0000","last_fullsized":"2024-09-13T12:02:03.544429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":790528,"data_stored":628382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":184320,"data_stored":85988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":96632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":91850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":97896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":690994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:08.884 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:09.204 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:09.205 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:09.216 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T12:02:07.562434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":449664,"kb_used_data":4780,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509499776,"statfs":{"total":1546188226560,"available":1545727770624,"internally_reserved":0,"allocated":4894720,"data_stored":2645441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.325780+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:03.325780+0000","last_peered":"2024-09-13T12:02:03.325780+0000","last_clean":"2024-09-13T12:02:03.325780+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:03.325780+0000","last_undegraded":"2024-09-13T12:02:03.325780+0000","last_fullsized":"2024-09-13T12:02:03.325780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.322894+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:03.322894+0000","last_peered":"2024-09-13T12:02:03.322894+0000","last_clean":"2024-09-13T12:02:03.322894+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:03.322894+0000","last_undegraded":"2024-09-13T12:02:03.322894+0000","last_fullsized":"2024-09-13T12:02:03.322894+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.358118+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:03.358118+0000","last_peered":"2024-09-13T12:02:03.358118+0000","last_clean":"2024-09-13T12:02:03.358118+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:03.358118+0000","last_undegraded":"2024-09-13T12:02:03.358118+0000","last_fullsized":"2024-09-13T12:02:03.358118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357832+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:03.357832+0000","last_peered":"2024-09-13T12:02:03.357832+0000","last_clean":"2024-09-13T12:02:03.357832+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:03.357832+0000","last_undegraded":"2024-09-13T12:02:03.357832+0000","last_fullsized":"2024-09-13T12:02:03.357832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.328998+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:03.328998+0000","last_peered":"2024-09-13T12:02:03.328998+0000","last_clean":"2024-09-13T12:02:03.328998+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:03.328998+0000","last_undegraded":"2024-09-13T12:02:03.328998+0000","last_fullsized":"2024-09-13T12:02:03.328998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357586+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:03.357586+0000","last_peered":"2024-09-13T12:02:03.357586+0000","last_clean":"2024-09-13T12:02:03.357586+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:03.357586+0000","last_undegraded":"2024-09-13T12:02:03.357586+0000","last_fullsized":"2024-09-13T12:02:03.357586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:04.493017+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:04.493017+0000","last_peered":"2024-09-13T12:02:04.493017+0000","last_clean":"2024-09-13T12:02:04.493017+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:04.493017+0000","last_undegraded":"2024-09-13T12:02:04.493017+0000","last_fullsized":"2024-09-13T12:02:04.493017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357455+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:03.357455+0000","last_peered":"2024-09-13T12:02:03.357455+0000","last_clean":"2024-09-13T12:02:03.357455+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:03.357455+0000","last_undegraded":"2024-09-13T12:02:03.357455+0000","last_fullsized":"2024-09-13T12:02:03.357455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.544429+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:03.544429+0000","last_peered":"2024-09-13T12:02:03.544429+0000","last_clean":"2024-09-13T12:02:03.544429+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:03.544429+0000","last_undegraded":"2024-09-13T12:02:03.544429+0000","last_fullsized":"2024-09-13T12:02:03.544429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":790528,"data_stored":628382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":184320,"data_stored":85988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":96632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":91850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":97896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":690994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:09.218 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T12:02:09.218 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T12:02:09.218 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T12:02:09.218 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T12:02:09.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:09.554 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T12:02:09.567 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T12:02:09.567 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T12:02:09.577 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T12:02:09.577 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'ms type': 'async', 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T12:02:09.578 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-13T12:02:09.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T12:02:09.900 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T12:02:09.900 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T12:02:09.900 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:09.992 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:09.991+0000 7f3291cf2640 -1 osd.13 28 handle_osd_map injecting map crc failure 2024-09-13T12:02:10.198 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:10.199 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:10.212 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T12:02:10.001352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":449664,"kb_used_data":4780,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509499776,"statfs":{"total":1546188226560,"available":1545727770624,"internally_reserved":0,"allocated":4894720,"data_stored":2645441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.439445"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.325780+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:03.325780+0000","last_peered":"2024-09-13T12:02:03.325780+0000","last_clean":"2024-09-13T12:02:03.325780+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:03.325780+0000","last_undegraded":"2024-09-13T12:02:03.325780+0000","last_fullsized":"2024-09-13T12:02:03.325780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.322894+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:03.322894+0000","last_peered":"2024-09-13T12:02:03.322894+0000","last_clean":"2024-09-13T12:02:03.322894+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:03.322894+0000","last_undegraded":"2024-09-13T12:02:03.322894+0000","last_fullsized":"2024-09-13T12:02:03.322894+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.358118+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:03.358118+0000","last_peered":"2024-09-13T12:02:03.358118+0000","last_clean":"2024-09-13T12:02:03.358118+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:03.358118+0000","last_undegraded":"2024-09-13T12:02:03.358118+0000","last_fullsized":"2024-09-13T12:02:03.358118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357832+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:03.357832+0000","last_peered":"2024-09-13T12:02:03.357832+0000","last_clean":"2024-09-13T12:02:03.357832+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:03.357832+0000","last_undegraded":"2024-09-13T12:02:03.357832+0000","last_fullsized":"2024-09-13T12:02:03.357832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.328998+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:03.328998+0000","last_peered":"2024-09-13T12:02:03.328998+0000","last_clean":"2024-09-13T12:02:03.328998+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:03.328998+0000","last_undegraded":"2024-09-13T12:02:03.328998+0000","last_fullsized":"2024-09-13T12:02:03.328998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357586+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:03.357586+0000","last_peered":"2024-09-13T12:02:03.357586+0000","last_clean":"2024-09-13T12:02:03.357586+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:03.357586+0000","last_undegraded":"2024-09-13T12:02:03.357586+0000","last_fullsized":"2024-09-13T12:02:03.357586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:04.493017+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:04.493017+0000","last_peered":"2024-09-13T12:02:04.493017+0000","last_clean":"2024-09-13T12:02:04.493017+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:04.493017+0000","last_undegraded":"2024-09-13T12:02:04.493017+0000","last_fullsized":"2024-09-13T12:02:04.493017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357455+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:03.357455+0000","last_peered":"2024-09-13T12:02:03.357455+0000","last_clean":"2024-09-13T12:02:03.357455+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:03.357455+0000","last_undegraded":"2024-09-13T12:02:03.357455+0000","last_fullsized":"2024-09-13T12:02:03.357455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.544429+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:03.544429+0000","last_peered":"2024-09-13T12:02:03.544429+0000","last_clean":"2024-09-13T12:02:03.544429+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:03.544429+0000","last_undegraded":"2024-09-13T12:02:03.544429+0000","last_fullsized":"2024-09-13T12:02:03.544429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":790528,"data_stored":628382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":184320,"data_stored":85988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":96632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":91850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":97896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":690994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:10.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:10.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:10.513 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:10.525 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T12:02:10.001352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":449664,"kb_used_data":4780,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509499776,"statfs":{"total":1546188226560,"available":1545727770624,"internally_reserved":0,"allocated":4894720,"data_stored":2645441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.439445"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.325780+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:03.325780+0000","last_peered":"2024-09-13T12:02:03.325780+0000","last_clean":"2024-09-13T12:02:03.325780+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:03.325780+0000","last_undegraded":"2024-09-13T12:02:03.325780+0000","last_fullsized":"2024-09-13T12:02:03.325780+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.322894+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:03.322894+0000","last_peered":"2024-09-13T12:02:03.322894+0000","last_clean":"2024-09-13T12:02:03.322894+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:03.322894+0000","last_undegraded":"2024-09-13T12:02:03.322894+0000","last_fullsized":"2024-09-13T12:02:03.322894+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.358118+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:03.358118+0000","last_peered":"2024-09-13T12:02:03.358118+0000","last_clean":"2024-09-13T12:02:03.358118+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:03.358118+0000","last_undegraded":"2024-09-13T12:02:03.358118+0000","last_fullsized":"2024-09-13T12:02:03.358118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357832+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:03.357832+0000","last_peered":"2024-09-13T12:02:03.357832+0000","last_clean":"2024-09-13T12:02:03.357832+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:03.357832+0000","last_undegraded":"2024-09-13T12:02:03.357832+0000","last_fullsized":"2024-09-13T12:02:03.357832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.328998+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:03.328998+0000","last_peered":"2024-09-13T12:02:03.328998+0000","last_clean":"2024-09-13T12:02:03.328998+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:03.328998+0000","last_undegraded":"2024-09-13T12:02:03.328998+0000","last_fullsized":"2024-09-13T12:02:03.328998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357586+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:03.357586+0000","last_peered":"2024-09-13T12:02:03.357586+0000","last_clean":"2024-09-13T12:02:03.357586+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:03.357586+0000","last_undegraded":"2024-09-13T12:02:03.357586+0000","last_fullsized":"2024-09-13T12:02:03.357586+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:04.493017+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:04.493017+0000","last_peered":"2024-09-13T12:02:04.493017+0000","last_clean":"2024-09-13T12:02:04.493017+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:04.493017+0000","last_undegraded":"2024-09-13T12:02:04.493017+0000","last_fullsized":"2024-09-13T12:02:04.493017+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.357455+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:03.357455+0000","last_peered":"2024-09-13T12:02:03.357455+0000","last_clean":"2024-09-13T12:02:03.357455+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:03.357455+0000","last_undegraded":"2024-09-13T12:02:03.357455+0000","last_fullsized":"2024-09-13T12:02:03.357455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T12:02:03.544429+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:03.544429+0000","last_peered":"2024-09-13T12:02:03.544429+0000","last_clean":"2024-09-13T12:02:03.544429+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:03.544429+0000","last_undegraded":"2024-09-13T12:02:03.544429+0000","last_fullsized":"2024-09-13T12:02:03.544429+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":790528,"data_stored":628382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247813,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":184320,"data_stored":85988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":96632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":91850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":97253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":91953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247815,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":93913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":97896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":93353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":856064,"data_stored":690994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":95293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:10.526 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T12:02:10.526 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-13T12:02:10.823 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 29 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 0694c701-d7eb-494e-ad86-64faa4809c32 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-09-13T12:01:45.016372+0000 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:modified 2024-09-13T12:02:09.990378+0000 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:crush_version 6 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:full_ratio 0.95 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:backfillfull_ratio 0.9 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:nearfull_ratio 0.85 2024-09-13T12:02:10.824 INFO:teuthology.orchestra.run.smithi005.stdout:require_min_compat_client luminous 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:min_compat_client jewel 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:require_osd_release squid 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:stretch_mode_enabled false 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.00 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:max_osd 16 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6804/3436814551 v2:172.21.15.5:6805/3436814551 exists,up 17c551c2-db05-4b58-af49-885510ae31ae 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6808/1772774403 v2:172.21.15.37:6809/1772774403 exists,up a19df369-47ed-4f8c-a422-f009d11c2360 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6804/4035060935 v2:172.21.15.39:6805/4035060935 exists,up 990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/4253545661 v2:172.21.15.87:6810/4253545661 exists,up 6b725dee-2e37-4fd1-a123-4d38d0e6979e 2024-09-13T12:02:10.825 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6812/1078609236 v2:172.21.15.5:6813/1078609236 exists,up 1b99fff1-9eb6-4883-bc7d-7422a005750d 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6812/702787639 v2:172.21.15.37:6813/702787639 exists,up 148d447f-4841-4fe1-8051-5f3f0094e5d1 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 up in weight 1 up_from 23 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6808/3381071726 v2:172.21.15.39:6809/3381071726 exists,up ac4ec297-ccbb-4bbe-94b3-4859b1a7851a 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6805/607675738 v2:172.21.15.87:6806/607675738 exists,up f8b5938e-5b74-4ef8-bb86-508b56a05296 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6808/608390337 v2:172.21.15.5:6809/608390337 exists,up 407a9a27-7549-489e-8a12-b0951ac5dc26 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6804/3323352543 v2:172.21.15.37:6805/3323352543 exists,up 4f199848-02c4-43ac-8344-36cd6dac6034 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6800/157727954 v2:172.21.15.39:6801/157727954 exists,up dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6801/1031207741 v2:172.21.15.87:6802/1031207741 exists,up 52c5aaa9-47d2-41da-a8f2-bb1c69f90259 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6800/2604688698 v2:172.21.15.5:6801/2604688698 exists,up 96321c3f-1589-4749-ba88-b48cdd72140d 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6800/3207086675 v2:172.21.15.37:6801/3207086675 exists,up 4942dad7-9e60-41a8-8098-12de4684b4f1 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6812/3204532746 v2:172.21.15.39:6813/3204532746 exists,up 1fa4c21c-7e24-49bb-9065-f8a189b9676b 2024-09-13T12:02:10.826 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6813/2774283835 v2:172.21.15.87:6814/2774283835 exists,up aba98af9-0ce7-4292-97fc-f15df5e45ddb 2024-09-13T12:02:10.836 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6804/3436814551 v2:172.21.15.5:6805/3436814551 exists,up 17c551c2-db05-4b58-af49-885510ae31ae', 'osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6808/1772774403 v2:172.21.15.37:6809/1772774403 exists,up a19df369-47ed-4f8c-a422-f009d11c2360', 'osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6804/4035060935 v2:172.21.15.39:6805/4035060935 exists,up 990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e', 'osd.3 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/4253545661 v2:172.21.15.87:6810/4253545661 exists,up 6b725dee-2e37-4fd1-a123-4d38d0e6979e', 'osd.4 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6812/1078609236 v2:172.21.15.5:6813/1078609236 exists,up 1b99fff1-9eb6-4883-bc7d-7422a005750d', 'osd.5 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6812/702787639 v2:172.21.15.37:6813/702787639 exists,up 148d447f-4841-4fe1-8051-5f3f0094e5d1', 'osd.6 up in weight 1 up_from 23 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6808/3381071726 v2:172.21.15.39:6809/3381071726 exists,up ac4ec297-ccbb-4bbe-94b3-4859b1a7851a', 'osd.7 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6805/607675738 v2:172.21.15.87:6806/607675738 exists,up f8b5938e-5b74-4ef8-bb86-508b56a05296', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6808/608390337 v2:172.21.15.5:6809/608390337 exists,up 407a9a27-7549-489e-8a12-b0951ac5dc26', 'osd.9 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6804/3323352543 v2:172.21.15.37:6805/3323352543 exists,up 4f199848-02c4-43ac-8344-36cd6dac6034', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6800/157727954 v2:172.21.15.39:6801/157727954 exists,up dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b', 'osd.11 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6801/1031207741 v2:172.21.15.87:6802/1031207741 exists,up 52c5aaa9-47d2-41da-a8f2-bb1c69f90259', 'osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.5:6800/2604688698 v2:172.21.15.5:6801/2604688698 exists,up 96321c3f-1589-4749-ba88-b48cdd72140d', 'osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.37:6800/3207086675 v2:172.21.15.37:6801/3207086675 exists,up 4942dad7-9e60-41a8-8098-12de4684b4f1', 'osd.14 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.39:6812/3204532746 v2:172.21.15.39:6813/3204532746 exists,up 1fa4c21c-7e24-49bb-9065-f8a189b9676b', 'osd.15 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6813/2774283835 v2:172.21.15.87:6814/2774283835 exists,up aba98af9-0ce7-4292-97fc-f15df5e45ddb'] 2024-09-13T12:02:10.836 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-13T12:02:10.938 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:02:10.938 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "mon.a", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster": "ceph", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "admin_socket_mode": "", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_client_required": "cephx, none", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_cluster_required": "cephx", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_debug": "true", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_required": "cephx", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "auth_supported": "cephx", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio": "true", 2024-09-13T12:02:10.939 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_aio_reap_max": "16", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_async_discard_threads": "0", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_block_size": "4096", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio": "false", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_enable_discard": "false", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry": "3", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T12:02:10.940 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash": "0", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring": "false", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_hipri": "false", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bdev_type": "", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_allocator": "hybrid", 2024-09-13T12:02:10.941 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_buffered_io": "true", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T12:02:10.942 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery": "false", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluefs_sync_write": "false", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_allocator": "hybrid", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T12:02:10.943 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_create": "true", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_create": "false", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_path": "", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_db_size": "0", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_path": "", 2024-09-13T12:02:10.944 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_size": "107374182400", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_create": "false", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_path": "", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs": "true", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T12:02:10.945 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune": "true", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size": "0", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cache_type": "2q", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T12:02:10.946 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_clone_cow": "true", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_mode": "none", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T12:02:10.947 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_freelist": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T12:02:10.948 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fail_eio": "false", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T12:02:10.949 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T12:02:10.950 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size": "0", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T12:02:10.951 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T12:02:10.952 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_spdk_mem": "512", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T12:02:10.953 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_tracing": "false", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_write_v2": "false", 2024-09-13T12:02:10.954 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_write_v2_random": "false", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T12:02:10.955 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_cluster_require_version": "2", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_signatures": "false", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_require_version": "2", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_signatures": "false", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_service_require_version": "2", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "cephx_sign_messages": "true", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "chdir": "", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "client_acl_type": "", 2024-09-13T12:02:10.956 INFO:teuthology.orchestra.run.smithi005.stdout: "client_asio_thread_count": "2", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_mid": "0.750000", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_cache_size": "16384", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_caps_release_delay": "5", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_check_pool_perm": "true", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_force_sync_read": "false", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_getattr_caps": "false", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_inject_features": "", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_die_on_failed_remount": "false", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_dirsize_rbytes": "true", 2024-09-13T12:02:10.957 INFO:teuthology.orchestra.run.smithi005.stdout: "client_force_lazyio": "false", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_fs": "", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_inject_release_failure": "false", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_inline_size": "4096", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mds_namespace": "", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_metadata": "", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_gid": "-1", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_timeout": "300", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mount_uid": "-1", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_mountpoint": "/", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_notify_timeout": "10", 2024-09-13T12:02:10.958 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc": "true", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_max_objects": "1000", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_size": "209715200", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_permissions": "true", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota": "true", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_quota_df": "true", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_bytes": "0", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_max_periods": "4", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_readahead_min": "131072", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_reconnect_stale": "false", 2024-09-13T12:02:10.959 INFO:teuthology.orchestra.run.smithi005.stdout: "client_shutdown_timeout": "30", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_snapdir": ".snap", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_tick_interval": "1", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_trace": "", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_faked_inos": "false", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "client_use_random_mds": "false", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog": "false", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_graylog_port": "12201", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_monitors": "default=true", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog": "false", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "clog_to_syslog_level": "info", 2024-09-13T12:02:10.960 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_addr": "-", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network": "", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "cluster_network_interface": "", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_isal": "false", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_level": "5", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "compressor_zstd_level": "1", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T12:02:10.961 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location": "", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook": "", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "crush_location_hook_timeout": "10", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "daemonize": "false", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "daos_pool": "tank", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_alienstore": "0/5", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok": "1/5", 2024-09-13T12:02:10.962 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asok_assert_abort": "false", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_auth": "20/20", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bdev": "1/3", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluefs": "1/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_bluestore": "1/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_buffer": "0/1", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cephsqlite": "0/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_client": "0/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_compressor": "1/5", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_context": "0/1", 2024-09-13T12:02:10.963 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crush": "1/1", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_crypto": "1/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_cyanstore": "0/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_dpdk": "1/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_eventtrace": "1/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filer": "0/1", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_filestore": "1/3", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_finisher": "1/1", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_fuse": "1/5", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T12:02:10.964 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_javaclient": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journal": "1/3", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_journaler": "0/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_kstore": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_lockdep": "0/1", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mclock": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_balancer": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_locker": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_migrator": "1/5", 2024-09-13T12:02:10.965 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_memstore": "1/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgr": "2/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mgrc": "1/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_mon": "20/20", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_monc": "0/10", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_ms": "1/1", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_none": "0/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objclass": "0/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objectcacher": "0/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_objecter": "0/1", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_optracker": "0/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_osd": "1/5", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_paxos": "20/20", 2024-09-13T12:02:10.966 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_perfcounter": "1/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_prioritycache": "1/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rados": "0/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd": "0/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rbd_replay": "0/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_refs": "0/0", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_reserver": "1/1", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw": "1/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_access": "1/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T12:02:10.967 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_flight": "1/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_notification": "1/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rgw_sync": "1/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_rocksdb": "4/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_backref": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cache": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_device": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_epm": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T12:02:10.968 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_journal": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_lba": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_odata": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_omap": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_onode": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_t": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_seastore_tm": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_striper": "0/1", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_test": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_throttle": "1/1", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_timer": "0/1", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_tp": "0/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "debug_trace": "1/5", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T12:02:10.969 INFO:teuthology.orchestra.run.smithi005.stdout: "device_failure_prediction_mode": "none", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_graylog": "false", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_journald": "false", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_stderr": "true", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "err_to_syslog": "false", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "event_tracing": "false", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_cert_file": "", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_http_port": "9926", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_key_file": "", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_prio_limit": "5", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T12:02:10.970 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_sort_metrics": "true", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "exporter_stats_period": "5", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "fake_statfs_for_testing": "0", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "fatal_signal_handlers": "true", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_purge_ops": "10", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filer_max_truncate_ops": "128", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_blackhole": "false", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_btrfs_snap": "true", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_caller_concurrency": "10", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T12:02:10.971 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_omap_check": "false", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_debug_verify_split": "false", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_dump_file": "", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fadvise": "true", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fail_eio": "true", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fd_cache_size": "128", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap": "false", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T12:02:10.972 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_inject_stall": "0", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_parallel": "false", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_trailing": "false", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_journal_writeahead": "false", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_kill_at": "0", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T12:02:10.973 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_merge_threshold": "-10", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_odsync_write": "false", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T12:02:10.974 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_backend_path": "", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_op_threads": "2", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_punch_hole": "false", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T12:02:10.975 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_queue_max_ops": "50", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_seek_data_hole": "false", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc": "false", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_splice": "false", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_multiple": "2", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_split_rand_factor": "20", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_update_to": "1000", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T12:02:10.976 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_xfs_extsize": "false", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "filestore_zfs_snap": "false", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "fio_dir": "/tmp/fio", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "fsid": "0694c701-d7eb-494e-ad86-64faa4809c32", 2024-09-13T12:02:10.977 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_allow_other": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_big_writes": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_debug": "false", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_default_permissions": "false", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_disable_pagecache": "false", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_max_write": "0", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_multithreaded": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_require_active_mds": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_set_user_groups": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_move": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_read": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_splice_write": "true", 2024-09-13T12:02:10.978 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "gss_target_name": "ceph", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_file": "", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_inject_failure": "0", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "heartbeat_interval": "5", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "host": "smithi005", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T12:02:10.979 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "inject_early_sigterm": "false", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "jaeger_agent_port": "6799", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "jaeger_tracing_enable": "false", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_aio": "true", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_align_min_size": "65536", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_align": "true", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_block_size": "4096", 2024-09-13T12:02:10.980 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_dio": "true", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_discard": "false", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_force_aio": "false", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_ignore_corruption": "false", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_max_write_entries": "100", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_replay_from": "0", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_write_header_frequency": "0", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journal_zero_on_create": "false", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prefetch_periods": "10", 2024-09-13T12:02:10.981 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_prezero_periods": "5", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "journaler_write_head_interval": "15", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "keyfile": "", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_backend": "rocksdb", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_bytes": "67108864", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_max_ops": "512", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_onode_map_size": "1024", 2024-09-13T12:02:10.982 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "kstore_sync_transaction": "false", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "librados_thread_count": "2", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep": "false", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "lockdep_force_backtrace": "false", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_coarse_timestamps": "true", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_flush_on_exit": "false", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_graylog_port": "12201", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_new": "1000", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_max_recent": "10000", 2024-09-13T12:02:10.983 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stderr_prefix": "", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_file": "true", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_graylog": "false", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_journald": "false", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_stderr": "false", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "log_to_syslog": "false", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_action_on_write_error": "1", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_alternate_name_max": "8192", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_asio_thread_count": "2", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_export_pin": "true", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T12:02:10.984 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_interval": "10", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max": "-1", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_max_until": "-1", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_merge_size": "50", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_mode": "0", 2024-09-13T12:02:10.985 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_bits": "3", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_size": "10000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T12:02:10.986 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_mid": "0.700000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_interval": "1", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T12:02:10.987 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_auth_pins": "false", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_frag": "false", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_scatterstat": "false", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_debug_subtrees": "false", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_default_dir_hash": "2", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_defer_session_stale": "true", 2024-09-13T12:02:10.988 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_max_entries": "0", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dir_prefetch": "true", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_early_reply": "true", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enable_op_tracker": "true", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_enforce_unique_name": "true", 2024-09-13T12:02:10.989 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T12:02:10.990 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_health_dummy": "false", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_join_fs": "", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_journal_format": "1", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_create_at": "0", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_export_at": "0", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_import_at": "0", 2024-09-13T12:02:10.991 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_link_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_openc_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_rename_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_events": "-1", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_max_segments": "128", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_pause": "false", 2024-09-13T12:02:10.992 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_segment_size": "0", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_threshold": "128", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_completed_requests": "100000", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_export_size": "20971520", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_file_recover": "32", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_files": "64", 2024-09-13T12:02:10.993 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops": "8192", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_metrics_update_interval": "2", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_per_client": "100", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_numa_node": "-1", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_duration": "600", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_size": "20", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T12:02:10.994 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_op_log_threshold": "5", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_grace": "15", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_ping_interval": "5", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_caps": "30000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_interval": "1.000000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T12:02:10.995 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_gid": "0", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_root_ino_uid": "0", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T12:02:10.996 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_shutdown_check": "0", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_skip_ino": "0", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_min_uid": "0", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_snap_rstat": "false", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_symlink_recovery": "true", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_exports": "0", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_thrash_fragments": "0", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_tick_interval": "5.000000", 2024-09-13T12:02:10.997 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_valgrind_exit": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_backtrace": "1", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_verify_scatter": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mds_wipe_sessions": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mempool_debug": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_set": "false", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "memstore_page_size": "65536", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_bytes": "134217728", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_messages": "512", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T12:02:10.998 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_disabled_modules": "", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mds_messages": "128", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_mon_messages": "128", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T12:02:10.999 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_history_duration": "600", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_history_size": "20", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_op_log_threshold": "5", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_osd_messages": "8192", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_pool": "true", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_standby_modules": "true", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_period": "5", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_stats_threshold": "5", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_tick_period": "2", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T12:02:11.000 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_delete": "true", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_allow_pool_size_one": "true", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_bytes": "104857600", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_directed_command_retry": "5", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T12:02:11.001 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_client_target_rank": "-1", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_level": "debug", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T12:02:11.002 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_start": "false", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_compact_on_trim": "true", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_cpu_threads": "4", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T12:02:11.003 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_crit": "5", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_avail_warn": "5", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_json": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_extra_checks": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T12:02:11.004 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_down_mkfs_grace": "120", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_down_uptime_grace": "60", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_default_strategy": "1", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_election_timeout": "5.000000", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_enable_op_tracker": "true", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_fake_pool_delete": "false", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_force_quorum_join": "false", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T12:02:11.005 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_detail_to_clog": "false", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_log_update_period": "5", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_max_detail": "50", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog": "true", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host": "172.21.15.5,172.21.15.37,172.21.15.39", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_host_override": "", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_initial_members": "", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T12:02:11.006 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease": "5.000000", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_full_interval": "50", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max": "10000", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_log_max_summary": "50", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_log_epochs": "500", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_osd": "10000", 2024-09-13T12:02:11.007 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_autotune": "true", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_memory_target": "2147483648", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_digest_period": "5", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T12:02:11.008 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_complaint_time": "30", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_duration": "600", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_size": "20", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_op_log_threshold": "5", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T12:02:11.009 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size": "500", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T12:02:11.010 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_report_timeout": "900", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T12:02:11.011 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T12:02:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_max_osds": "4", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_interval": "300", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_max_keys": "100", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_scrub_timeout": "300", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_session_timeout": "300", 2024-09-13T12:02:11.013 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_smart_report_timeout": "5", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_pool_size": "4", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_debug": "false", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T12:02:11.014 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_tick_interval": "5", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T12:02:11.015 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_older_version": "true", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "monmap": "", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T12:02:11.016 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "motr_tracing_enabled": "false", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_op_threads": "3", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_cm": "false", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_device_name": "", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T12:02:11.017 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_sl": "3", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_rdma_type": "ib", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_async_reap_threshold": "5", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_before_connect": "false", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv4": "true", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_ipv6": "false", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr1": "false", 2024-09-13T12:02:11.018 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_msgr2": "true", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_max": "7568", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_port_min": "6800", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_count": "3", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_bind_retry_delay": "5", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_client": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mds": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mgr": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_mon": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_blackhole_osd": "false", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_client_mode": "crc secure", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T12:02:11.019 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_cluster_type": "", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_compress_secure": "false", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_data": "true", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_crc_header": "true", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_bug": "true", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_old_message": "true", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T12:02:11.020 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hugepages": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_lro": "true", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_pmd": "", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_port_id": "0", 2024-09-13T12:02:11.021 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_dump_on_send": "false", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_delay_type": "", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_network_congestion": "0", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_inject_socket_failures": "2500", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_accept_failures": "4", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_max_backoff": "15.000000", 2024-09-13T12:02:11.022 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compress_mode": "none", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_pq_min_cost": "65536", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_public_type": "", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_service_mode": "crc secure", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_nodelay": "true", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T12:02:11.023 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "ms_type": "async", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "no_config_file": "false", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inflight_ops": "1024", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objecter_timeout": "10.000000", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objectstore_blackhole": "false", 2024-09-13T12:02:11.024 INFO:teuthology.orchestra.run.smithi005.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "openssl_engine_opts": "", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_max_ops": "4", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_agent_slop": "0.020000", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_asio_thread_count": "2", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_async_recovery_min_cost": "1", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T12:02:11.025 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_max": "512", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backfill_scan_min": "64", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_peering": "false", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_beacon_report_interval": "300", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_duration": "30", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_blkin_trace_all": "false", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T12:02:11.026 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_class_update_on_start": "true", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_cap": "256", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_op_priority": "63", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_client_watch_timeout": "30", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_command_max_records": "256", 2024-09-13T12:02:11.027 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_compact_on_start": "false", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_on_start": "true", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T12:02:11.028 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_op_order": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_shutdown": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_trim_objects": "false", 2024-09-13T12:02:11.029 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_default_notify_timeout": "30", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T12:02:11.030 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_ec_partial_reads": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_enable_op_tracker": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_info": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T12:02:11.031 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_function_tracing": "false", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_grace": "20", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_interval": "6", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_stale": "600", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T12:02:11.032 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_journal_size": "5120", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_kill_backfill_at": "0", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_cache_size": "50", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_dedup": "true", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max": "40", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T12:02:11.033 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_name_len": "100", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_attr_size": "0", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_backfills": "1", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_count": "5", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_markdown_period": "600", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_name_len": "2048", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_object_size": "134217728", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T12:02:11.034 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_pgls": "1024", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_cost": "8388608", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_push_objects": "10", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_scrubs": "3", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_max_write_size": "90", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T12:02:11.035 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_profile": "balanced", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T12:02:11.036 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_base": "805306368", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target": "4294967296", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_autotune": "false", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_min_recovery_priority": "0", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_interval": "5", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T12:02:11.037 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_cache_shards": "32", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_node": "-1", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objecter_finishers": "1", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore": "bluestore", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_fuse": "false", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_objectstore_tracing": "false", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_duration": "600", 2024-09-13T12:02:11.038 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_size": "20", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_log_threshold": "5", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards": "0", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T12:02:11.039 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_op_thread_timeout": "15", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_open_classes_on_start": "true", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_os_flags": "0", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_peering_op_priority": "255", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_delete_priority": "5", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T12:02:11.040 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_crimson": "false", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T12:02:11.041 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_flags": "0", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_min_size": "2", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_size": "2", 2024-09-13T12:02:11.042 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_default_type": "replicated", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_cost": "20971520", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active": "0", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T12:02:11.043 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_priority": "3", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_priority": "5", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_repair_during_recovery": "false", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T12:02:11.044 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_cost": "52428800", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_hour": "0", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T12:02:11.045 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_priority": "5", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T12:02:11.046 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_skip_data_digest": "false", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_smart_report_timeout": "5", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_priority": "5", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_target_transaction_size": "30", 2024-09-13T12:02:11.047 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_tracing": "false", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_use_stale_snap": "false", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T12:02:11.048 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_kill_at": "0", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_max_join_drift": "10", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min": "500", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_min_wait": "0.050000", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max": "500", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_service_trim_min": "10", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_stash_full_interval": "25", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_max": "500", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "paxos_trim_min": "250", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "perf": "true", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T12:02:11.049 INFO:teuthology.orchestra.run.smithi005.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addr": "-", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_addrv": "", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_bind_addr": "-", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network": "", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "public_network_interface": "", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_enabled": "false", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_mon_op_timeout": "0", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_osd_op_timeout": "0", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rados_tracing": "false", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_atime_update_interval": "60", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T12:02:11.050 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache": "true", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_policy": "writearound", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_size": "33554432", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T12:02:11.051 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_compression_hint": "none", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_clone_format": "auto", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_data_pool": "", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_features": "61", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_format": "2", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_map_options": "", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_order": "22", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_pool": "rbd", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_count": "0", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T12:02:11.052 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler": "simple", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T12:02:11.053 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_order": "24", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_pool": "", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_journal_splay_width": "4", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T12:02:11.054 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T12:02:11.055 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T12:02:11.056 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_op_threads": "1", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_plugins": "", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T12:02:11.057 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T12:02:11.058 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_tracing": "false", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_names": "true", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rbd_validate_pool": "true", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_base_url": "", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "restapi_log_level": "", 2024-09-13T12:02:11.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_admin_entry": "admin", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_backend_store": "rados", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_barbican_url": "", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_beast_enable_async": "true", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T12:02:11.060 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_enabled": "true", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_config_store": "rados", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_content_length_compat": "false", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T12:02:11.061 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_cross_domain_policy": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T12:02:11.062 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T12:02:11.063 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T12:02:11.064 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T12:02:11.065 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_log_window": "30", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T12:02:11.066 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_disable_s3select": "false", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T12:02:11.067 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_name": "", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dns_s3website_name": "", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_ops_log": "false", 2024-09-13T12:02:11.068 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_static_website": "false", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enable_usage_log": "false", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_expose_bucket": "false", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_extended_http_attrs": "", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_filter": "none", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T12:02:11.069 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_objs": "32", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_init_timeout": "300", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T12:02:11.070 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_project": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_admin_user": "", 2024-09-13T12:02:11.071 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_url": "", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T12:02:11.072 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_objs": "32", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_worker": "3", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ldap_uri": "", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T12:02:11.073 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_http_headers": "", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attr_size": "0", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T12:02:11.074 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_listing_results": "1000", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_notify_retries": "10", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T12:02:11.075 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T12:02:11.076 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_num_control_oids": "8", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_numa_node": "-1", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_op_tracing": "false", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_token": "", 2024-09-13T12:02:11.077 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_url": "", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_rados": "false", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T12:02:11.078 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_continue": "true", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rados_tracing": "false", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm": "", 2024-09-13T12:02:11.079 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_id": "", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region": "", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_request_uri": "", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T12:02:11.080 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_resolve_cname": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_run_sync_thread": "true", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T12:02:11.081 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_script_uri": "", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_service_provider_name": "", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_id": "", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_client_secret": "", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_entry": "sts", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_key": "sts", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T12:02:11.082 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_auth_url": "", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_custom_header": "", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_need_stats": "true", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_tenant_name": "", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url": "", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_lease_period": "120", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T12:02:11.083 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_thread_pool_size": "512", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_comment": "", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_createby": "", 2024-09-13T12:02:11.084 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_encoding": "", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_flag": "false", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_origin": "", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_torrent_tracker": "", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_shards": "32", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_use_opa_authz": "false", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_counters_cache": "false", 2024-09-13T12:02:11.085 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_user_unique_email": "true", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_verify_ssl": "true", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone": "", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_id": "", 2024-09-13T12:02:11.086 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup": "", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_id": "", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_block_size": "4096", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T12:02:11.087 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_partition_filters": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_perf": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T12:02:11.088 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "run_dir": "/var/run/ceph", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_block_create": "true", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_device_size": "53687091200", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_full_integrity_check": "false", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T12:02:11.089 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_main_device_type": "SSD", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "seastore_segment_size": "67108864", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "set_keepcaps": "false", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "setgroup": "", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser": "", 2024-09-13T12:02:11.090 INFO:teuthology.orchestra.run.smithi005.stdout: "setuser_match_path": "", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "thp": "false", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_default_timeout": "60", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "throttler_perf_counter": "true", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "tmp_dir": "/tmp", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "uadk_compressor_enabled": "false", 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T12:02:11.091 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:02:11.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T12:02:11.211 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-09-13T12:02:11.212 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:11.221 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-09-13T12:02:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:11.231 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-09-13T12:02:11.231 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:11.242 DEBUG:teuthology.orchestra.run.smithi087:> type ceph-objectstore-tool 2024-09-13T12:02:11.260 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T12:02:11.260 DEBUG:teuthology.orchestra.run.smithi005:> type ceph-objectstore-tool 2024-09-13T12:02:11.275 INFO:teuthology.orchestra.run.smithi005.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T12:02:11.275 DEBUG:teuthology.orchestra.run.smithi037:> type ceph-objectstore-tool 2024-09-13T12:02:11.294 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T12:02:11.294 DEBUG:teuthology.orchestra.run.smithi039:> type ceph-objectstore-tool 2024-09-13T12:02:11.313 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T12:02:11.313 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T12:02:11.328 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T12:02:11.328 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:02:11.328 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create base 4' 2024-09-13T12:02:11.329 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T12:02:11.329 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T12:02:11.329 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T12:02:11.329 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-13T12:02:11.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-13T12:02:11.329 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T12:02:11.330 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T12:02:11.330 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T12:02:11.330 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T12:02:11.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T12:02:11.331 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T12:02:11.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T12:02:11.354 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:11.354+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:11.431 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:11.429+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:11.450 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:02:11.451 INFO:teuthology.orchestra.run.smithi005.stdout: "ops": [], 2024-09-13T12:02:11.451 INFO:teuthology.orchestra.run.smithi005.stdout: "num_ops": 0 2024-09-13T12:02:11.451 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:02:11.462 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T12:02:11.473 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-13T12:02:11.474 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T12:02:11.490 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:11.490 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:11.490 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:11.490 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:11.491 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:11.493 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-13T12:02:11.493 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T12:02:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:11.495 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:11.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-13T12:02:11.512 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-13T12:02:11.512 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T12:02:11.544 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-13T12:02:11.545 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T12:02:11.545 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:11.543+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:11.581 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-13T12:02:11.581 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T12:02:11.586 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:11.587 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:11.596 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T12:02:11.617 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-13T12:02:11.617 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T12:02:11.633 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:11.632+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:11.636 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-13T12:02:11.636 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T12:02:11.656 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:11.656 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:11.656 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:11.656 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:11.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-13T12:02:11.658 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T12:02:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:11.660 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:11.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-13T12:02:11.674 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-13T12:02:11.674 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T12:02:11.692 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-13T12:02:11.692 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T12:02:11.710 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-13T12:02:11.710 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T12:02:11.726 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-13T12:02:11.727 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T12:02:11.732 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:11.732 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:11.734 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:11.734+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:11.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T12:02:11.744 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-13T12:02:11.744 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T12:02:11.775 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-13T12:02:11.775 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T12:02:11.792 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-13T12:02:11.792 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T12:02:11.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:11.809+0000 7f24bad9d640 0 -- 172.21.15.5:0/3822939682 >> v2:172.21.15.87:6813/2774283835 conn(0x7f2494033470 msgr2=0x7f2494035860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T12:02:11.811 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-13T12:02:11.811 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:11.828 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:11.829 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:11.829 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:11.829 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:11.831 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:11.832 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-13T12:02:11.857 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:11.857 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:11.857 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:11.855+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:11.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T12:02:11.936 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:11.936+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.016 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-13T12:02:12.028 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.028 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.037 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:12.036+0000 7ff4f559a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33463) UID: 0 2024-09-13T12:02:12.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T12:02:12.138 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:12.137+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.155 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.156 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.156 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.156 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.158 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.158 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-13T12:02:12.187 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.188 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.196 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T12:02:12.238 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:12.237+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.301 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.301 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-13T12:02:12.339 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:12.339+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:12.347 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.347 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T12:02:12.372 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-13T12:02:12.375 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'base' created 2024-09-13T12:02:12.386 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:12.385+0000 7f63ff314640 -1 osd.14 29 handle_osd_map injecting map crc failure 2024-09-13T12:02:12.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool application enable base rados' 2024-09-13T12:02:12.440 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:12.438+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.450 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.451 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.453 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-13T12:02:12.469 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.470 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.478 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T12:02:12.540 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:12.539+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:12.637 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.637 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.637 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.638 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.640 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 'trim stale osdmaps' 2024-09-13T12:02:12.642 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:12.641+0000 7f8c9f769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33460) UID: 0 2024-09-13T12:02:12.647 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.647 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.656 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T12:02:12.743 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:12.742+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.780 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.781 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.784 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 'trim stale osdmaps' 2024-09-13T12:02:12.807 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.807 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T12:02:12.843 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:12.843+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:12.945 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:12.944+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:12.947 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:12.947 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:12.948 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:12.951 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:12.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 'trim stale osdmaps' 2024-09-13T12:02:12.967 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:12.967 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:12.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T12:02:13.045 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:13.045+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.091 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.092 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.092 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.094 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.094 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 'trim stale osdmaps' 2024-09-13T12:02:13.126 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.126 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.135 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T12:02:13.177 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:13.175+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.232 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.233 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.235 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.236 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 'trim stale osdmaps' 2024-09-13T12:02:13.256 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.256 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.265 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T12:02:13.277 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:13.276+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:13.348 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:13.346+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:13.373 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.373 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rados' on pool 'base' 2024-09-13T12:02:13.382 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T12:02:13.394 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-09-13T12:02:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.414 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.415 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.418 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 'trim stale osdmaps' 2024-09-13T12:02:13.449 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:13.448+0000 7f8c9f769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33460) UID: 0 2024-09-13T12:02:13.464 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:13.462+0000 7f36821c9640 -1 osd.11 30 handle_osd_map injecting map crc failure 2024-09-13T12:02:13.491 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.491 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.499 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T12:02:13.550 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:13.550+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.559 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.559 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 'trim stale osdmaps' 2024-09-13T12:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.649 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.651 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:13.650+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:13.657 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T12:02:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.696 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.700 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 'trim stale osdmaps' 2024-09-13T12:02:13.752 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:13.751+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:13.806 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.806 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.815 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T12:02:13.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:no valid command found; 10 closest matches: 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:0 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:1 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:2 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:abort 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:assert 2024-09-13T12:02:13.840 INFO:teuthology.orchestra.run.smithi005.stderr:bench [] [] [] [] 2024-09-13T12:02:13.841 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs debug_inject_read_zeros 2024-09-13T12:02:13.841 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs files list 2024-09-13T12:02:13.841 INFO:teuthology.orchestra.run.smithi005.stderr:bluefs stats 2024-09-13T12:02:13.841 INFO:teuthology.orchestra.run.smithi005.stderr:bluestore allocator dump block 2024-09-13T12:02:13.841 INFO:teuthology.orchestra.run.smithi005.stderr:Error EINVAL: invalid command 2024-09-13T12:02:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T12:02:13.843 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-13T12:02:13.856 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:13.855+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:13.950 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-13T12:02:13.951 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T12:02:13.953 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:13.953+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:13.965 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:13.965 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:13.968 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-13T12:02:13.968 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T12:02:13.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T12:02:13.985 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-13T12:02:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T12:02:14.003 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-13T12:02:14.003 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T12:02:14.019 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-13T12:02:14.019 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T12:02:14.036 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-13T12:02:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T12:02:14.052 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-13T12:02:14.053 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T12:02:14.054 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:14.053+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:14.085 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:14.085 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:14.094 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T12:02:14.155 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:14.155+0000 7fbe495a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33152) UID: 0 2024-09-13T12:02:14.202 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:14.202 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:14.210 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T12:02:14.256 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:14.255+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:14.271 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-13T12:02:14.271 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T12:02:14.287 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-13T12:02:14.287 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T12:02:14.304 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-13T12:02:14.304 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T12:02:14.318 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:14.318 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:14.318 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:14.318 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [] 2024-09-13T12:02:14.318 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:14.321 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-13T12:02:14.321 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T12:02:14.327 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T12:02:14.338 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-13T12:02:14.339 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T12:02:14.354 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-13T12:02:14.355 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T12:02:14.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:14.355+0000 7f008e59c640 0 -- 172.21.15.5:0/3170975149 >> v2:172.21.15.37:6800/3207086675 conn(0x7f009021ea90 msgr2=0x7f0090220e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T12:02:14.356 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:14.355+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:14.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-13T12:02:14.372 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T12:02:14.387 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cache' created 2024-09-13T12:02:14.390 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-13T12:02:14.390 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T12:02:14.394 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T12:02:14.402 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add base cache' 2024-09-13T12:02:14.407 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-13T12:02:14.407 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T12:02:14.435 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:14.435 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:14.435 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-13T12:02:14.435 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:14.443 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T12:02:14.488 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:14.487+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:14.551 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:14.551 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:14.551 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-13T12:02:14.551 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-13T12:02:14.551 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:14.558 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:14.557+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:14.560 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T12:02:14.659 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:14.657+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:14.687 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:14.685+0000 7f4dca859640 -1 osd.15 31 handle_osd_map injecting map crc failure 2024-09-13T12:02:14.708 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:14.708 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:14.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:14.708 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.100743043, 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.049861256, 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 2 2024-09-13T12:02:14.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607025+0000", 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1749999999999994e-06 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607034+0000", 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4930000000000004e-06 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607047+0000", 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2731999999999999e-05 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607071+0000", 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4287e-05 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607152+0000", 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0820000000000002e-05 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for peered", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607160+0000", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6399999999999997e-06 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.656733+0000", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.049573503999999997 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.656877+0000", 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014360499999999999 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.713 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.050631580999999, 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039068399999999997, 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:14.714 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 3 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657128+0000", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1069999999999999e-06 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657134+0000", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5720000000000002e-06 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657144+0000", 2024-09-13T12:02:14.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8649999999999996e-06 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657150+0000", 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4740000000000002e-06 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657386+0000", 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023575000000000001 2024-09-13T12:02:14.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657518+0000", 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013191599999999999 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-13T12:02:14.718 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.380843431000001, 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020399189999999998, 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 5 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326917+0000", 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4440000000000001e-06 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326924+0000", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.627e-06 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326939+0000", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4713e-05 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326948+0000", 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.098e-06 2024-09-13T12:02:14.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327013+0000", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4568999999999998e-05 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327129+0000", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000116119 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for subops from 12", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327241+0000", 2024-09-13T12:02:14.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011259100000000001 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op_commit", 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328302+0000", 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001060533 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328846+0000", 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054400399999999995 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-13T12:02:14.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328910+0000", 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4021000000000002e-05 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328955+0000", 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5200000000000001e-05 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop_reply(client.4329.0:5 2.2 e27/25 ondisk, result = 0)", 2024-09-13T12:02:14.724 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 12.379008074, 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000225498, 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328752+0000", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.153e-06 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328756+0000", 2024-09-13T12:02:14.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7210000000000002e-06 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328768+0000", 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1987000000000001e-05 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328775+0000", 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.601e-06 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328828+0000", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3059000000000001e-05 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328839+0000", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1382e-05 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328976+0000", 2024-09-13T12:02:14.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013659499999999999 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.384393227, 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005684283, 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.729 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 7 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323367+0000", 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.945e-06 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323378+0000", 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1042e-05 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323391+0000", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2502e-05 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323401+0000", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8900000000000002e-06 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323483+0000", 2024-09-13T12:02:14.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2014000000000005e-05 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323490+0000", 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3030000000000002e-06 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325558+0000", 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020676029999999999 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325628+0000", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0043999999999995e-05 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for subops from 12", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325748+0000", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012014399999999999 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327246+0000", 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014984040000000001 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op_commit", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.328927+0000", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00168053 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.328995+0000", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8643e-05 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.329050+0000", 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4218999999999997e-05 2024-09-13T12:02:14.735 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop_reply(client.4329.0:7 2.2 e27/25 ondisk, result = 0)", 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 11.380616873999999, 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000110201, 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:14.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:14.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327143+0000", 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1299999999999998e-07 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327148+0000", 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1760000000000001e-06 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327160+0000", 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2439000000000001e-05 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327167+0000", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4919999999999998e-06 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327232+0000", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5259000000000007e-05 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327239+0000", 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7089999999999996e-06 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:14.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327252+0000", 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3213e-05 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:14.740 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:14.741 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T12:02:14.760 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:14.759+0000 7ff4f559a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33463) UID: 0 2024-09-13T12:02:14.861 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:14.860+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:14.862 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:14.862 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:14.862 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:14.862 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:14.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T12:02:14.962 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:14.961+0000 7f5b94d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33741) UID: 0 2024-09-13T12:02:15.021 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.021 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:15.021 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:15.022 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:15.022 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T12:02:15.063 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:15.062+0000 7f5b94d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33741) UID: 0 2024-09-13T12:02:15.137 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.137 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:15.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:15.137 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-13T12:02:15.137 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T12:02:15.164 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:15.163+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:15.253 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:15.253 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.261 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T12:02:15.264 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:15.264+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:15.365 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:15.364+0000 7f4dd5e58640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33118) UID: 0 2024-09-13T12:02:15.367 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.367 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:15.368 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:15.368 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:15.368 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T12:02:15.385 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is set 2024-09-13T12:02:15.386 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'cache' is now (or already was) a tier of 'base' 2024-09-13T12:02:15.401 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-09-13T12:02:15.465 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:15.464+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:15.486 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.486 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:15.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.178655588000002, 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.24283714300000001, 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.487 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305654+0000", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4099999999999998e-06 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305669+0000", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5403000000000001e-05 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305693+0000", 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3085999999999999e-05 2024-09-13T12:02:15.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305710+0000", 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305833+0000", 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012282699999999999 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-13T12:02:15.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305839+0000", 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5509999999999999e-06 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.306949+0000", 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011098460000000001 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.306956+0000", 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1709999999999996e-06 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.546985+0000", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.24002928400000001 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547132+0000", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014670899999999999 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547279+0000", 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000147594 2024-09-13T12:02:15.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547876+0000", 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059653899999999999 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548422+0000", 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00054650499999999997 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548458+0000", 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5840999999999998e-05 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548489+0000", 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0434e-05 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:15.494 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616547690999999, 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022150008999999998, 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 95 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:15.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2600000000000001e-07 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867762+0000", 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7940000000000001e-06 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867766+0000", 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1409999999999998e-06 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867771+0000", 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1139999999999997e-06 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881288+0000", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013517662 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881331+0000", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2508e-05 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881411+0000", 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0037000000000006e-05 2024-09-13T12:02:15.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884636+0000", 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032248789999999999 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889838+0000", 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052020570000000004 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889868+0000", 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0241000000000001e-05 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889910+0000", 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1350000000000002e-05 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.500 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616532084999999, 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022304243000000001, 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 96 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2600000000000001e-07 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867839+0000", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3635999999999999e-05 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867846+0000", 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3599999999999998e-06 2024-09-13T12:02:15.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867863+0000", 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6240999999999999e-05 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883515+0000", 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015652454 2024-09-13T12:02:15.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883590+0000", 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5239000000000002e-05 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883718+0000", 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000128107 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889963+0000", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062445590000000002 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889989+0000", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6222000000000001e-05 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890039+0000", 2024-09-13T12:02:15.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9934e-05 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890079+0000", 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0265000000000002e-05 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.507 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616428784, 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022308126000000001, 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 97 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.508 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867879+0000", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0699999999999998e-07 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867882+0000", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.785e-06 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867886+0000", 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1559999999999997e-06 2024-09-13T12:02:15.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867891+0000", 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9579999999999998e-06 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884670+0000", 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016778899 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884712+0000", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1900000000000002e-05 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884785+0000", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3382999999999999e-05 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890097+0000", 2024-09-13T12:02:15.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053122270000000001 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890122+0000", 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4964e-05 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890148+0000", 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5935000000000002e-05 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890187+0000", 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8512e-05 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616411691, 2024-09-13T12:02:15.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026179832, 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 98 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:15.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4699999999999998e-07 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867898+0000", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.838e-06 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867901+0000", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9849999999999998e-06 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867903+0000", 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7269999999999999e-06 2024-09-13T12:02:15.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886608+0000", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018704498999999999 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886697+0000", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8950999999999996e-05 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886826+0000", 2024-09-13T12:02:15.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893970+0000", 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0071436410000000001 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893983+0000", 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3303e-05 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894031+0000", 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8424000000000003e-05 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894075+0000", 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4011000000000003e-05 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616400184, 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026267485, 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 99 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.521 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.36e-07 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867945+0000", 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7614000000000003e-05 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867950+0000", 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5219999999999998e-06 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867954+0000", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1750000000000001e-06 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890237+0000", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022283857000000001 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890311+0000", 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3528000000000005e-05 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890453+0000", 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142292 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894085+0000", 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036321000000000001 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894110+0000", 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4430000000000002e-05 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894137+0000", 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.705e-05 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894174+0000", 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7681000000000001e-05 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.616347384000001, 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026761970999999999, 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.527 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 100 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5699999999999998e-07 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867962+0000", 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0389999999999999e-06 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867965+0000", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0730000000000001e-06 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867968+0000", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6699999999999998e-06 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893588+0000", 2024-09-13T12:02:15.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025619583000000001 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893633+0000", 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5175e-05 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893704+0000", 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1187999999999997e-05 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894299+0000", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059460600000000002 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894636+0000", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033761700000000002 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894687+0000", 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1243999999999999e-05 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894722+0000", 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4418999999999997e-05 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:15.533 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512824267999999, 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020011478999999999, 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:15.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8100000000000003e-07 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971573+0000", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9506999999999995e-05 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971579+0000", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5289999999999997e-06 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971583+0000", 2024-09-13T12:02:15.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5839999999999999e-06 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.982929+0000", 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011345815 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.983008+0000", 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8930999999999994e-05 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.983133+0000", 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125473 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991353+0000", 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082199360000000006 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991400+0000", 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6857000000000002e-05 2024-09-13T12:02:15.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991450+0000", 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0074999999999997e-05 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991494+0000", 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4291e-05 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:118 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.51271713, 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020014107, 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 118 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.540 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.41e-07 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971593+0000", 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2079999999999999e-06 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971596+0000", 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0309999999999999e-06 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971598+0000", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6979999999999998e-06 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985629+0000", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014030447 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985673+0000", 2024-09-13T12:02:15.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3739000000000003e-05 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985743+0000", 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0772999999999995e-05 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991506+0000", 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005762217 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991553+0000", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7089000000000003e-05 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991583+0000", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0417e-05 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991604+0000", 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1146999999999999e-05 2024-09-13T12:02:15.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:119 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512702942000001, 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020132049999999999, 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.546 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 119 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:15.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971605+0000", 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4999999999999999e-07 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971607+0000", 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6060000000000001e-06 2024-09-13T12:02:15.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971611+0000", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.574e-06 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971613+0000", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4899999999999999e-06 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986173+0000", 2024-09-13T12:02:15.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014559458000000001 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986214+0000", 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1353999999999997e-05 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986284+0000", 2024-09-13T12:02:15.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0315999999999996e-05 2024-09-13T12:02:15.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991511+0000", 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052270270000000004 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991649+0000", 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013772400000000001 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991696+0000", 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7258000000000001e-05 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991736+0000", 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9993000000000001e-05 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:120 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.553 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512684952000001, 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022538208000000001, 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 120 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971623+0000", 2024-09-13T12:02:15.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4299999999999999e-07 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971676+0000", 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3770999999999997e-05 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971682+0000", 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9089999999999997e-06 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971685+0000", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2310000000000001e-06 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988715+0000", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017029279000000001 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988793+0000", 2024-09-13T12:02:15.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8590999999999993e-05 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988928+0000", 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013418699999999999 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991779+0000", 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002851115 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994073+0000", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022939380000000001 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994118+0000", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5383000000000002e-05 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994160+0000", 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2460999999999999e-05 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.559 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:121 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.51261506, 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026118208, 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.560 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 121 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:15.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5100000000000001e-07 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971694+0000", 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9300000000000002e-06 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971698+0000", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0879999999999999e-06 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971708+0000", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0319000000000001e-05 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991816+0000", 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020108595 2024-09-13T12:02:15.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991898+0000", 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1909000000000005e-05 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.992029+0000", 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013046999999999999 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994603+0000", 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025743430000000002 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997718+0000", 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031146350000000001 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997764+0000", 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6567999999999999e-05 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997810+0000", 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6e-05 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512590577999999, 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026284658999999998, 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.566 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 122 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.84e-07 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971719+0000", 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.322e-06 2024-09-13T12:02:15.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971723+0000", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4989999999999999e-06 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971727+0000", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5510000000000001e-06 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994207+0000", 2024-09-13T12:02:15.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022479642000000001 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994264+0000", 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6489000000000001e-05 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994336+0000", 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2243999999999995e-05 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997856+0000", 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035200769999999999 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997917+0000", 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1638999999999997e-05 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997962+0000", 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4165000000000002e-05 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998001+0000", 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9647000000000002e-05 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512545661000001, 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026342913999999999, 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.572 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 123 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9699999999999996e-07 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971845+0000", 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2752999999999999e-05 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971859+0000", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4265e-05 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971867+0000", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1219999999999995e-06 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994628+0000", 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022760493 2024-09-13T12:02:15.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994670+0000", 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2063e-05 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994742+0000", 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1843999999999999e-05 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997904+0000", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031624470000000001 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998013+0000", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108989 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998057+0000", 2024-09-13T12:02:15.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3504999999999998e-05 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998104+0000", 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7836000000000001e-05 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.512426872000001, 2024-09-13T12:02:15.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026341582999999998, 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 124 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:15.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971881+0000", 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4099999999999999e-07 2024-09-13T12:02:15.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971886+0000", 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2769999999999996e-06 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971893+0000", 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4999999999999996e-06 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971898+0000", 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4410000000000001e-06 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995230+0000", 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023332037 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995310+0000", 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9889999999999996e-05 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995439+0000", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012856599999999999 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998124+0000", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026858540000000001 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998137+0000", 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2875e-05 2024-09-13T12:02:15.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998183+0000", 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5945000000000001e-05 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998222+0000", 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8656999999999999e-05 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.584 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808901718, 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023141144999999998, 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 138 2024-09-13T12:02:15.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675406+0000", 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7999999999999996e-07 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675539+0000", 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013268899999999999 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675543+0000", 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7010000000000004e-06 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675547+0000", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2890000000000002e-06 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681389+0000", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058419350000000004 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681434+0000", 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5762e-05 2024-09-13T12:02:15.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681502+0000", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7429999999999996e-05 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687572+0000", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060703909999999996 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698492+0000", 2024-09-13T12:02:15.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010919816000000001 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698524+0000", 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1982e-05 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698547+0000", 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2670000000000001e-05 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808753279999999, 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023142126999999998, 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.591 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 139 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2099999999999998e-07 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675556+0000", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0779999999999998e-06 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675559+0000", 2024-09-13T12:02:15.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9409999999999999e-06 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675562+0000", 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5749999999999999e-06 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687731+0000", 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012169377 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687775+0000", 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3971000000000003e-05 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687843+0000", 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7547999999999998e-05 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698593+0000", 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010750048 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698644+0000", 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1555999999999999e-05 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698669+0000", 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4426999999999998e-05 2024-09-13T12:02:15.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698696+0000", 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7285e-05 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808741294000001, 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034648480000000002, 2024-09-13T12:02:15.597 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 140 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5899999999999998e-07 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675569+0000", 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3249999999999999e-06 2024-09-13T12:02:15.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675572+0000", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9210000000000001e-06 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675576+0000", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3680000000000002e-06 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693053+0000", 2024-09-13T12:02:15.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017477650000000001 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693120+0000", 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6754999999999994e-05 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693227+0000", 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106557 2024-09-13T12:02:15.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698635+0000", 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054081279999999999 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710168+0000", 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011532804000000001 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710194+0000", 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5935000000000002e-05 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710214+0000", 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0778e-05 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.603 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808727518, 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034677564000000001, 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 141 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3900000000000001e-07 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675629+0000", 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8739000000000002e-05 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675634+0000", 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1290000000000004e-06 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675638+0000", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2889999999999999e-06 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698747+0000", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023109333999999999 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698800+0000", 2024-09-13T12:02:15.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2583000000000001e-05 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698883+0000", 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2655000000000001e-05 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710129+0000", 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011246493999999999 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710219+0000", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9802999999999994e-05 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710239+0000", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0531999999999999e-05 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710257+0000", 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7767000000000001e-05 2024-09-13T12:02:15.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:142 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.808661892, 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034777052000000003, 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:15.610 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 142 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:15.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675646+0000", 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3500000000000002e-07 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675648+0000", 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0490000000000002e-06 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:15.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675652+0000", 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9709999999999998e-06 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675654+0000", 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4990000000000001e-06 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704497+0000", 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028843131000000001 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704548+0000", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0553000000000002e-05 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704634+0000", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6497999999999998e-05 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710162+0000", 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055272610000000003 2024-09-13T12:02:15.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710360+0000", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000198075 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710399+0000", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9165999999999997e-05 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710422+0000", 2024-09-13T12:02:15.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3513999999999999e-05 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:15.616 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T12:02:15.618 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:15.617+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:15.718 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:15.716+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:15.765 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.765 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:15.765 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:15.766 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T12:02:15.775 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:15.774+0000 7f4db32f9640 -1 osd.0 32 handle_osd_map injecting map crc failure 2024-09-13T12:02:15.819 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:15.818+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:15.880 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:15.880 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:15.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T12:02:15.920 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:15.919+0000 7f4dd5e58640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33118) UID: 0 2024-09-13T12:02:15.995 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:15.995 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:15.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:15.995 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-13T12:02:15.995 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:16.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T12:02:16.021 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:16.020+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:16.110 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:16.111 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:16.111 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:16.111 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:16.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T12:02:16.121 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:16.120+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:16.222 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:16.221+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:16.225 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:16.225 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:16.225 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:16.226 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:16.226 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:16.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T12:02:16.323 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:16.322+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4329.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:01.306862+0000", 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.033802903, 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.29632206500000002, 2024-09-13T12:02:16.341 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4329", 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306862+0000", 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:16.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306862+0000", 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306865+0000", 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8160000000000004e-06 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:16.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306873+0000", 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4710000000000001e-06 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306883+0000", 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0118e-05 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306893+0000", 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5049999999999996e-06 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306988+0000", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5890000000000005e-05 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.306994+0000", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.755e-06 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.315890+0000", 2024-09-13T12:02:16.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0088956020000000007 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.315896+0000", 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4359999999999999e-06 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.601612+0000", 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.28571590899999999 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.601738+0000", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012568900000000001 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 5", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.601889+0000", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015120800000000001 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.602610+0000", 2024-09-13T12:02:16.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00072062600000000004 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603117+0000", 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00050695899999999997 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603154+0000", 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7418e-05 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603184+0000", 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9663e-05 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop_reply(client.4329.0:1 2.3 e26/25 ondisk, result = 0)", 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:01.603028+0000", 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.737636063, 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016185099999999999, 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:16.349 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603028+0000", 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603028+0000", 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:16.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603029+0000", 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7499999999999999e-07 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603036+0000", 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3949999999999998e-06 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:16.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603049+0000", 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3253e-05 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603058+0000", 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2410000000000001e-06 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603104+0000", 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5982999999999997e-05 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603112+0000", 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8699999999999992e-06 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:01.603190+0000", 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8233999999999997e-05 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:16.353 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:16.354 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:16.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T12:02:16.395 INFO:teuthology.orchestra.run.smithi087.stderr:set cache-mode for pool 'cache' to writeback 2024-09-13T12:02:16.411 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay base cache' 2024-09-13T12:02:16.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T12:02:16.424 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:16.424+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:16.465 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-13T12:02:16.465 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-09-13T12:02:16.465 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-09-13T12:02:16.465 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-13T12:02:16.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T12:02:16.525 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:16.524+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:16.556 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-13T12:02:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T12:02:16.574 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-13T12:02:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T12:02:16.591 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-13T12:02:16.592 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T12:02:16.610 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-13T12:02:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T12:02:16.621 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:16.621 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:16.625 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:16.625+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:16.626 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-13T12:02:16.626 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T12:02:16.630 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T12:02:16.643 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-13T12:02:16.644 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T12:02:16.661 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-13T12:02:16.661 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T12:02:16.679 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-13T12:02:16.679 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T12:02:16.695 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-13T12:02:16.695 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T12:02:16.712 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-13T12:02:16.712 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T12:02:16.726 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:16.725+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:16.729 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-13T12:02:16.729 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T12:02:16.736 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:16.736 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:16.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T12:02:16.747 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-13T12:02:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T12:02:16.756 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:16.755+0000 7f322a927640 -1 osd.10 33 handle_osd_map injecting map crc failure 2024-09-13T12:02:16.764 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-13T12:02:16.764 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T12:02:16.782 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-13T12:02:16.782 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T12:02:16.799 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-13T12:02:16.799 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T12:02:16.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:16.799+0000 7fa151563640 0 -- 172.21.15.5:0/627797935 >> v2:172.21.15.87:6813/2774283835 conn(0x7fa14c2247a0 msgr2=0x7fa14c226b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T12:02:16.829 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:16.828+0000 7f83e195f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33110) UID: 0 2024-09-13T12:02:16.889 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:16.889 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:16.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T12:02:16.938 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:16.937+0000 7f83e195f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33110) UID: 0 2024-09-13T12:02:17.030 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:17.030+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:17.040 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.041 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T12:02:17.131 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:17.130+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:17.152 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.152 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.161 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T12:02:17.232 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:17.231+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:17.264 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.265 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T12:02:17.333 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:17.332+0000 7f8c9f769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33460) UID: 0 2024-09-13T12:02:17.376 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.376 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.384 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T12:02:17.400 INFO:teuthology.orchestra.run.smithi087.stderr:overlay for 'base' is now (or already was) 'cache' 2024-09-13T12:02:17.404 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T12:02:17.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-09-13T12:02:17.420 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-13T12:02:17.420 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T12:02:17.434 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:17.433+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:17.491 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.491 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.500 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T12:02:17.535 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:17.535+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:17.635 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:17.634+0000 7f83e195f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33110) UID: 0 2024-09-13T12:02:17.654 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.654 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.664 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T12:02:17.737 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:17.736+0000 7fbe495a7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33152) UID: 0 2024-09-13T12:02:17.757 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:17.756+0000 7f374f3c5640 -1 osd.12 34 handle_osd_map injecting map crc failure 2024-09-13T12:02:17.806 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.806 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T12:02:17.838 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:17.837+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:17.917 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:17.917 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:17.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T12:02:17.939 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:17.938+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:18.035 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.035 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.040 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:18.038+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:18.043 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T12:02:18.140 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:18.139+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:18.151 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.151 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.160 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T12:02:18.242 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:18.241+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:18.307 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.307 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T12:02:18.360 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:18.359+0000 7f5b94d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33741) UID: 0 2024-09-13T12:02:18.412 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-13T12:02:18.413 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 hit_set_type to bloom 2024-09-13T12:02:18.425 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.425 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-09-13T12:02:18.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T12:02:18.443 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:18.442+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:18.545 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:18.543+0000 7f8c9f769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33460) UID: 0 2024-09-13T12:02:18.583 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.583 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.592 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T12:02:18.646 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:18.645+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:18.699 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.699 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T12:02:18.747 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:18.746+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:18.776 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:18.775+0000 7f8c90162640 -1 osd.5 34 handle_osd_map injecting map crc failure 2024-09-13T12:02:18.815 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.815 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.824 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T12:02:18.844 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T12:02:18.844 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T12:02:18.844 INFO:tasks.thrashosds.thrasher:inject_pause on osd.15 2024-09-13T12:02:18.844 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T12:02:18.845 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T12:02:18.845 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok config set filestore_inject_stall 3 2024-09-13T12:02:18.868 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:18.868+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:18.932 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:18.932 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:18.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T12:02:18.947 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-09-13T12:02:18.947 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T12:02:18.947 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-09-13T12:02:18.948 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:18.948+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:19.050 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:19.049+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:19.088 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:19.088 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:19.097 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T12:02:19.151 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:19.150+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:19.204 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:19.205 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:19.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T12:02:19.252 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:19.251+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:19.321 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:19.321 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:19.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T12:02:19.353 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:19.352+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:19.420 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:19.419+0000 7fc76752a640 -1 osd.4 36 handle_osd_map injecting map crc failure 2024-09-13T12:02:19.420 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 hit_set_count to 8 2024-09-13T12:02:19.420 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:19.419+0000 7f36841cd640 -1 osd.11 36 handle_osd_map injecting map crc failure 2024-09-13T12:02:19.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 60' 2024-09-13T12:02:19.437 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T12:02:19.438 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:19.438 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:19.446 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T12:02:19.488 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:19.487+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:19.554 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:19.553+0000 7ff4f559a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33463) UID: 0 2024-09-13T12:02:19.564 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-13T12:02:19.564 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T12:02:19.569 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:19.569 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:19.578 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T12:02:19.582 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-13T12:02:19.582 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T12:02:19.599 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-13T12:02:19.599 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T12:02:19.616 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-13T12:02:19.616 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T12:02:19.632 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-13T12:02:19.632 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T12:02:19.649 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-13T12:02:19.649 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T12:02:19.655 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:19.655+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:19.666 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-13T12:02:19.666 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T12:02:19.683 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-13T12:02:19.683 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T12:02:19.699 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-13T12:02:19.700 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T12:02:19.716 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-13T12:02:19.717 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T12:02:19.728 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:19.728 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:19.728 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-13T12:02:19.728 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-13T12:02:19.728 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:19.734 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-13T12:02:19.734 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T12:02:19.737 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T12:02:19.745 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:19.745+0000 7f4dca859640 -1 osd.15 35 handle_osd_map injecting map crc failure 2024-09-13T12:02:19.750 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-13T12:02:19.751 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T12:02:19.766 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-13T12:02:19.767 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T12:02:19.780 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:19.779+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:19.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-13T12:02:19.783 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T12:02:19.801 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-13T12:02:19.801 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T12:02:19.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:19.800+0000 7fb6614cf640 0 -- 172.21.15.5:0/3279134745 >> v2:172.21.15.87:6813/2774283835 conn(0x7fb65c224de0 msgr2=0x7fb65c2271d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T12:02:19.846 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:19.968 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:19.968 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:19.968 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [] 2024-09-13T12:02:19.969 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:19.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T12:02:19.971 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:19.969+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:20.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-13T12:02:20.020 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T12:02:20.071 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:20.070+0000 7f83e195f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33110) UID: 0 2024-09-13T12:02:20.118 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.119 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.119 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-13T12:02:20.119 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T12:02:20.172 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:20.171+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:20.234 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.234 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.234 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-13T12:02:20.234 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-13T12:02:20.234 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.242 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T12:02:20.273 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:20.272+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:20.350 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.350 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:20.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:20.350 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop(client.4329.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:01.602078+0000", 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.748067084999999, 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00089895800000000005, 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602078+0000", 2024-09-13T12:02:20.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602078+0000", 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602078+0000", 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0900000000000001e-07 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.352 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602085+0000", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.345e-06 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602101+0000", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6103000000000002e-05 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602111+0000", 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.713e-06 2024-09-13T12:02:20.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602156+0000", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5297999999999999e-05 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602183+0000", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6673999999999999e-05 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602925+0000", 2024-09-13T12:02:20.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00074219800000000001 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.602977+0000", 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1418000000000001e-05 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:20.355 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.359 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T12:02:20.402 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:20.401+0000 7f329f2f5640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33466) UID: 0 2024-09-13T12:02:20.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T12:02:20.432 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 hit_set_period to 60 2024-09-13T12:02:20.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache target_max_objects 250' 2024-09-13T12:02:20.466 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.466 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.466 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-13T12:02:20.466 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.474 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T12:02:20.476 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:20.475+0000 7f1acac3f640 -1 osd.1 37 handle_osd_map injecting map crc failure 2024-09-13T12:02:20.489 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:20.488+0000 7f4dd5e58640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33118) UID: 0 2024-09-13T12:02:20.574 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:20.574+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:20.583 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.583 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.583 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-13T12:02:20.583 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-13T12:02:20.583 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T12:02:20.675 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:20.674+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:20.698 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.698 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:20.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:20.698 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [] 2024-09-13T12:02:20.699 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.707 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T12:02:20.776 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:20.775+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:20.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-09-13T12:02:20.803+0000 7f7da37fe640 0 --2- 172.21.15.87:0/1841493439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7da411b3f0 0x7f7da411db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T12:02:20.855 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.855 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.856 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-09-13T12:02:20.856 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.866 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T12:02:20.876 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:20.876+0000 7fc774b2d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33738) UID: 0 2024-09-13T12:02:20.973 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-09-13T12:02:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-09-13T12:02:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-09-13T12:02:20.973 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:20.977 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:20.977+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:20.981 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T12:02:21.078 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:21.078+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:21.088 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-09-13T12:02:21.088 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.480881449999998, 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.049861256, 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:21.089 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 2 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607016+0000", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607025+0000", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1749999999999994e-06 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607034+0000", 2024-09-13T12:02:21.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4930000000000004e-06 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607047+0000", 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2731999999999999e-05 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607071+0000", 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4287e-05 2024-09-13T12:02:21.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607152+0000", 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0820000000000002e-05 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for peered", 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.607160+0000", 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6399999999999997e-06 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.093 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.656733+0000", 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.049573503999999997 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.656877+0000", 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014360499999999999 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.430769988000002, 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039068399999999997, 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 3 2024-09-13T12:02:21.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657127+0000", 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657128+0000", 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1069999999999999e-06 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657134+0000", 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5720000000000002e-06 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657144+0000", 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8649999999999996e-06 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657150+0000", 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4740000000000002e-06 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657386+0000", 2024-09-13T12:02:21.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023575000000000001 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:01.657518+0000", 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013191599999999999 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-13T12:02:21.099 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.760981837999999, 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020399189999999998, 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 5 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326915+0000", 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326917+0000", 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4440000000000001e-06 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326924+0000", 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.627e-06 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326939+0000", 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4713e-05 2024-09-13T12:02:21.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.326948+0000", 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.098e-06 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327013+0000", 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4568999999999998e-05 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:21.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327129+0000", 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000116119 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for subops from 12", 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.327241+0000", 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011259100000000001 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op_commit", 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328302+0000", 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001060533 2024-09-13T12:02:21.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328846+0000", 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054400399999999995 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328910+0000", 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4021000000000002e-05 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328955+0000", 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5200000000000001e-05 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop_reply(client.4329.0:5 2.2 e27/25 ondisk, result = 0)", 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:21.106 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.759146480999998, 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000225498, 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328751+0000", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328752+0000", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.153e-06 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328756+0000", 2024-09-13T12:02:21.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7210000000000002e-06 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328768+0000", 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1987000000000001e-05 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328775+0000", 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.601e-06 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328828+0000", 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3059000000000001e-05 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328839+0000", 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1382e-05 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:02.328976+0000", 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013659499999999999 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4329.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.764531634000001, 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.005684283, 2024-09-13T12:02:21.111 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4329", 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.5:0/2329595933", 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 7 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:21.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323365+0000", 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323367+0000", 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.945e-06 2024-09-13T12:02:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323378+0000", 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1042e-05 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323391+0000", 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2502e-05 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323401+0000", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8900000000000002e-06 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323483+0000", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2014000000000005e-05 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.323490+0000", 2024-09-13T12:02:21.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3030000000000002e-06 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325558+0000", 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0020676029999999999 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325628+0000", 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0043999999999995e-05 2024-09-13T12:02:21.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for subops from 12", 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.325748+0000", 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012014399999999999 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327246+0000", 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014984040000000001 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op_commit", 2024-09-13T12:02:21.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.328927+0000", 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00168053 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.328995+0000", 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8643e-05 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.329050+0000", 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4218999999999997e-05 2024-09-13T12:02:21.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_repop_reply(client.4329.0:7 2.2 e27/25 ondisk, result = 0)", 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.760755281000002, 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000110201, 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-09-13T12:02:21.119 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327142+0000", 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327143+0000", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1299999999999998e-07 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327148+0000", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1760000000000001e-06 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327160+0000", 2024-09-13T12:02:21.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2439000000000001e-05 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327167+0000", 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4919999999999998e-06 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327232+0000", 2024-09-13T12:02:21.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5259000000000007e-05 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327239+0000", 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7089999999999996e-06 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-09-13T12:02:03.327252+0000", 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3213e-05 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.123 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-09-13T12:02:21.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-09-13T12:02:21.124 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-09-13T12:02:21.124 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:21.111+0000 7f374f3c5640 -1 osd.12 37 handle_osd_map injecting map crc failure 2024-09-13T12:02:21.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T12:02:21.179 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:21.178+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:21.246 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.247 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:21.247 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:21.247 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:21.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T12:02:21.299 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:21.299+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:21.364 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:21.364 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:21.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T12:02:21.414 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:21.413+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:21.438 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-13T12:02:21.441 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 target_max_objects to 250 2024-09-13T12:02:21.456 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache min_read_recency_for_promote 2' 2024-09-13T12:02:21.481 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:21.480+0000 7f8c9f769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33460) UID: 0 2024-09-13T12:02:21.520 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.521 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:21.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:21.521 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-13T12:02:21.521 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:21.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T12:02:21.581 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:21.581+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:21.637 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.638 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:21.638 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:21.638 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:21.646 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T12:02:21.682 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:21.681+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:21.740 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:21.739+0000 7f36821c9640 -1 osd.11 37 handle_osd_map injecting map crc failure 2024-09-13T12:02:21.783 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:21.782+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:21.794 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:21.794 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:21.794 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:21.795 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:21.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T12:02:21.883 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:21.883+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:21.912 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.604944577000001, 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.24283714300000001, 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.913 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 1 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305652+0000", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305654+0000", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4099999999999998e-06 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305669+0000", 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5403000000000001e-05 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305693+0000", 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3085999999999999e-05 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305710+0000", 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305833+0000", 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012282699999999999 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.305839+0000", 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5509999999999999e-06 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.306949+0000", 2024-09-13T12:02:21.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011098460000000001 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for peered", 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.306956+0000", 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1709999999999996e-06 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.546985+0000", 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.24002928400000001 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547132+0000", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014670899999999999 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547279+0000", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000147594 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.547876+0000", 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059653899999999999 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548422+0000", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00054650499999999997 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548458+0000", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5840999999999998e-05 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.548489+0000", 2024-09-13T12:02:21.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0434e-05 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042836680000001, 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022150008999999998, 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.921 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 95 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867760+0000", 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2600000000000001e-07 2024-09-13T12:02:21.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867762+0000", 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7940000000000001e-06 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867766+0000", 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1409999999999998e-06 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867771+0000", 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1139999999999997e-06 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881288+0000", 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.013517662 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881331+0000", 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2508e-05 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.881411+0000", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0037000000000006e-05 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884636+0000", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032248789999999999 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889838+0000", 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052020570000000004 2024-09-13T12:02:21.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889868+0000", 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0241000000000001e-05 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889910+0000", 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1350000000000002e-05 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042821073999999, 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022304243000000001, 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 96 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.928 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867775+0000", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2600000000000001e-07 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867839+0000", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3635999999999999e-05 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867846+0000", 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3599999999999998e-06 2024-09-13T12:02:21.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867863+0000", 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6240999999999999e-05 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883515+0000", 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.015652454 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883590+0000", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5239000000000002e-05 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.883718+0000", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000128107 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889963+0000", 2024-09-13T12:02:21.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062445590000000002 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.889989+0000", 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6222000000000001e-05 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890039+0000", 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9934e-05 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890079+0000", 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0265000000000002e-05 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.934 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042717773, 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022308126000000001, 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 97 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.935 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867878+0000", 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867879+0000", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0699999999999998e-07 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867882+0000", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.785e-06 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867886+0000", 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1559999999999997e-06 2024-09-13T12:02:21.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867891+0000", 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9579999999999998e-06 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884670+0000", 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.016778899 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884712+0000", 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1900000000000002e-05 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.884785+0000", 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3382999999999999e-05 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890097+0000", 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053122270000000001 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890122+0000", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4964e-05 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890148+0000", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5935000000000002e-05 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890187+0000", 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8512e-05 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.940 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042700679999999, 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026179832, 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.941 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 98 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867896+0000", 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4699999999999998e-07 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867898+0000", 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.838e-06 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867901+0000", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9849999999999998e-06 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867903+0000", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7269999999999999e-06 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886608+0000", 2024-09-13T12:02:21.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.018704498999999999 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886697+0000", 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8950999999999996e-05 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.886826+0000", 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893970+0000", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0071436410000000001 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893983+0000", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3303e-05 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894031+0000", 2024-09-13T12:02:21.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8424000000000003e-05 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894075+0000", 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4011000000000003e-05 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.947 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042689172999999, 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026267485, 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 99 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867907+0000", 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.36e-07 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867945+0000", 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7614000000000003e-05 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867950+0000", 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5219999999999998e-06 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867954+0000", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1750000000000001e-06 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890237+0000", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022283857000000001 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890311+0000", 2024-09-13T12:02:21.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3528000000000005e-05 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.890453+0000", 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142292 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894085+0000", 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036321000000000001 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894110+0000", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4430000000000002e-05 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894137+0000", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.705e-05 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894174+0000", 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7681000000000001e-05 2024-09-13T12:02:21.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.042636373000001, 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026761970999999999, 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.954 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 100 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867960+0000", 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5699999999999998e-07 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867962+0000", 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0389999999999999e-06 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867965+0000", 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0730000000000001e-06 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.867968+0000", 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6699999999999998e-06 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893588+0000", 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025619583000000001 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893633+0000", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5175e-05 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.893704+0000", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1187999999999997e-05 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894299+0000", 2024-09-13T12:02:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059460600000000002 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894636+0000", 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033761700000000002 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894687+0000", 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1243999999999999e-05 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.894722+0000", 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4418999999999997e-05 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.939113256999999, 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020011478999999999, 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 117 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971483+0000", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8100000000000003e-07 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971573+0000", 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9506999999999995e-05 2024-09-13T12:02:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971579+0000", 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5289999999999997e-06 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971583+0000", 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5839999999999999e-06 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.982929+0000", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011345815 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.983008+0000", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8930999999999994e-05 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.983133+0000", 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125473 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991353+0000", 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082199360000000006 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991400+0000", 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6857000000000002e-05 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991450+0000", 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0074999999999997e-05 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991494+0000", 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4291e-05 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:118 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.939006118999998, 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020014107, 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 118 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971590+0000", 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.41e-07 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971593+0000", 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2079999999999999e-06 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971596+0000", 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0309999999999999e-06 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971598+0000", 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6979999999999998e-06 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985629+0000", 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014030447 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985673+0000", 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3739000000000003e-05 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.985743+0000", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0772999999999995e-05 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991506+0000", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005762217 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991553+0000", 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7089000000000003e-05 2024-09-13T12:02:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991583+0000", 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0417e-05 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991604+0000", 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1146999999999999e-05 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:119 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.938991931, 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020132049999999999, 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 119 2024-09-13T12:02:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971604+0000", 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971605+0000", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4999999999999999e-07 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971607+0000", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6060000000000001e-06 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971611+0000", 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.574e-06 2024-09-13T12:02:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971613+0000", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4899999999999999e-06 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986173+0000", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014559458000000001 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986214+0000", 2024-09-13T12:02:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.1353999999999997e-05 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.986284+0000", 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0315999999999996e-05 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991511+0000", 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052270270000000004 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991649+0000", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013772400000000001 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991696+0000", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7258000000000001e-05 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991736+0000", 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9993000000000001e-05 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:120 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.938973941, 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022538208000000001, 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 120 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971622+0000", 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971623+0000", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4299999999999999e-07 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971676+0000", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3770999999999997e-05 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971682+0000", 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9089999999999997e-06 2024-09-13T12:02:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971685+0000", 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2310000000000001e-06 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988715+0000", 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017029279000000001 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988793+0000", 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8590999999999993e-05 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.988928+0000", 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013418699999999999 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991779+0000", 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002851115 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994073+0000", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022939380000000001 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994118+0000", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5383000000000002e-05 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994160+0000", 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2460999999999999e-05 2024-09-13T12:02:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:121 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.938904049000001, 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026118208, 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 121 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971692+0000", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5100000000000001e-07 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971694+0000", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9300000000000002e-06 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971698+0000", 2024-09-13T12:02:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0879999999999999e-06 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971708+0000", 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0319000000000001e-05 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991816+0000", 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.020108595 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.991898+0000", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1909000000000005e-05 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.992029+0000", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013046999999999999 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994603+0000", 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025743430000000002 2024-09-13T12:02:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997718+0000", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031146350000000001 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997764+0000", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6567999999999999e-05 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997810+0000", 2024-09-13T12:02:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6e-05 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.938879567000001, 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026284658999999998, 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 122 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971717+0000", 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.84e-07 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971719+0000", 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.322e-06 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971723+0000", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4989999999999999e-06 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971727+0000", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5510000000000001e-06 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994207+0000", 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022479642000000001 2024-09-13T12:02:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994264+0000", 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6489000000000001e-05 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994336+0000", 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2243999999999995e-05 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997856+0000", 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035200769999999999 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997917+0000", 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1638999999999997e-05 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997962+0000", 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4165000000000002e-05 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998001+0000", 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9647000000000002e-05 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.93883465, 2024-09-13T12:02:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026342913999999999, 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 123 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971762+0000", 2024-09-13T12:02:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9699999999999996e-07 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971845+0000", 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2752999999999999e-05 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971859+0000", 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4265e-05 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971867+0000", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1219999999999995e-06 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994628+0000", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022760493 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994670+0000", 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2063e-05 2024-09-13T12:02:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.994742+0000", 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1843999999999999e-05 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.997904+0000", 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031624470000000001 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998013+0000", 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108989 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998057+0000", 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3504999999999998e-05 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998104+0000", 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7836000000000001e-05 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.938715860999999, 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026341582999999998, 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 124 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971880+0000", 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971881+0000", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4099999999999999e-07 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971886+0000", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2769999999999996e-06 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971893+0000", 2024-09-13T12:02:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4999999999999996e-06 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.971898+0000", 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4410000000000001e-06 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995230+0000", 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023332037 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995310+0000", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9889999999999996e-05 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.995439+0000", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012856599999999999 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998124+0000", 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026858540000000001 2024-09-13T12:02:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998137+0000", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2875e-05 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998183+0000", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5945000000000001e-05 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:01:59.998222+0000", 2024-09-13T12:02:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8656999999999999e-05 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.235190707000001, 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023141144999999998, 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 138 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675405+0000", 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675406+0000", 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7999999999999996e-07 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675539+0000", 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013268899999999999 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675543+0000", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7010000000000004e-06 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675547+0000", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2890000000000002e-06 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681389+0000", 2024-09-13T12:02:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058419350000000004 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681434+0000", 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5762e-05 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.681502+0000", 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7429999999999996e-05 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687572+0000", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060703909999999996 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698492+0000", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010919816000000001 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698524+0000", 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1982e-05 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698547+0000", 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2670000000000001e-05 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.235042269000001, 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023142126999999998, 2024-09-13T12:02:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 139 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675554+0000", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2099999999999998e-07 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675556+0000", 2024-09-13T12:02:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0779999999999998e-06 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675559+0000", 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9409999999999999e-06 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675562+0000", 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5749999999999999e-06 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687731+0000", 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.012169377 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687775+0000", 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3971000000000003e-05 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.687843+0000", 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7547999999999998e-05 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698593+0000", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.010750048 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698644+0000", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1555999999999999e-05 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698669+0000", 2024-09-13T12:02:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4426999999999998e-05 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698696+0000", 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7285e-05 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.235030283, 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034648480000000002, 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 140 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675566+0000", 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5899999999999998e-07 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675569+0000", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3249999999999999e-06 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675572+0000", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9210000000000001e-06 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675576+0000", 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3680000000000002e-06 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693053+0000", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.017477650000000001 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693120+0000", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6754999999999994e-05 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.693227+0000", 2024-09-13T12:02:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106557 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698635+0000", 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054081279999999999 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710168+0000", 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011532804000000001 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710194+0000", 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5935000000000002e-05 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710214+0000", 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0778e-05 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.235016507000001, 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034677564000000001, 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 141 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675580+0000", 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3900000000000001e-07 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675629+0000", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8739000000000002e-05 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675634+0000", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1290000000000004e-06 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675638+0000", 2024-09-13T12:02:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2889999999999999e-06 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698747+0000", 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023109333999999999 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698800+0000", 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2583000000000001e-05 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.698883+0000", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2655000000000001e-05 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710129+0000", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.011246493999999999 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710219+0000", 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9802999999999994e-05 2024-09-13T12:02:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710239+0000", 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0531999999999999e-05 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710257+0000", 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7767000000000001e-05 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4203.0:142 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.234950881, 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034777052000000003, 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4203", 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.87:0/2168979810", 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 142 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675645+0000", 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-13T12:02:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675646+0000", 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3500000000000002e-07 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675648+0000", 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0490000000000002e-06 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675652+0000", 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9709999999999998e-06 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.675654+0000", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4990000000000001e-06 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704497+0000", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028843131000000001 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704548+0000", 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0553000000000002e-05 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 15", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.704634+0000", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6497999999999998e-05 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710162+0000", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055272610000000003 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710360+0000", 2024-09-13T12:02:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000198075 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710399+0000", 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9165999999999997e-05 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-13T12:02:00.710422+0000", 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3513999999999999e-05 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-13T12:02:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-13T12:02:22.041 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:22.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T12:02:22.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T12:02:22.043 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:22.041+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:22.143 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:22.142+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:22.182 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0: {} 2024-09-13T12:02:22.183 INFO:teuthology.orchestra.run.smithi005.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T12:02:22.190 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:22.190 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-13T12:02:22.190 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:22.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T12:02:22.201 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1: {} 2024-09-13T12:02:22.201 INFO:teuthology.orchestra.run.smithi005.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T12:02:22.218 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2: {} 2024-09-13T12:02:22.218 INFO:teuthology.orchestra.run.smithi005.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T12:02:22.235 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3: {} 2024-09-13T12:02:22.235 INFO:teuthology.orchestra.run.smithi005.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T12:02:22.243 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:22.242+0000 7f5b94d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33741) UID: 0 2024-09-13T12:02:22.251 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4: {} 2024-09-13T12:02:22.251 INFO:teuthology.orchestra.run.smithi005.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T12:02:22.269 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5: {} 2024-09-13T12:02:22.269 INFO:teuthology.orchestra.run.smithi005.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T12:02:22.286 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6: {} 2024-09-13T12:02:22.286 INFO:teuthology.orchestra.run.smithi005.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T12:02:22.304 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7: {} 2024-09-13T12:02:22.304 INFO:teuthology.orchestra.run.smithi005.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T12:02:22.306 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-13T12:02:22.306 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-13T12:02:22.306 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-13T12:02:22.306 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-13T12:02:22.306 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-13T12:02:22.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T12:02:22.320 INFO:teuthology.orchestra.run.smithi005.stdout:osd.8: {} 2024-09-13T12:02:22.320 INFO:teuthology.orchestra.run.smithi005.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T12:02:22.337 INFO:teuthology.orchestra.run.smithi005.stdout:osd.9: {} 2024-09-13T12:02:22.337 INFO:teuthology.orchestra.run.smithi005.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T12:02:22.344 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:22.344+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:22.354 INFO:teuthology.orchestra.run.smithi005.stdout:osd.10: {} 2024-09-13T12:02:22.355 INFO:teuthology.orchestra.run.smithi005.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T12:02:22.372 INFO:teuthology.orchestra.run.smithi005.stdout:osd.11: {} 2024-09-13T12:02:22.372 INFO:teuthology.orchestra.run.smithi005.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T12:02:22.388 INFO:teuthology.orchestra.run.smithi005.stdout:osd.12: {} 2024-09-13T12:02:22.388 INFO:teuthology.orchestra.run.smithi005.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T12:02:22.405 INFO:teuthology.orchestra.run.smithi005.stdout:osd.13: {} 2024-09-13T12:02:22.405 INFO:teuthology.orchestra.run.smithi005.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T12:02:22.421 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:22.421 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:22.422 INFO:teuthology.orchestra.run.smithi005.stdout:osd.14: {} 2024-09-13T12:02:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T12:02:22.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:22.422+0000 7f98ded9d640 0 -- 172.21.15.5:0/1276105877 >> v2:172.21.15.87:6813/2774283835 conn(0x7f98e0224b10 msgr2=0x7f98e0226f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T12:02:22.430 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T12:02:22.457 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 min_read_recency_for_promote to 2 2024-09-13T12:02:22.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache min_write_recency_for_promote 2' 2024-09-13T12:02:22.473 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:22.473+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:22.537 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:22.537 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:22.545 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T12:02:22.557 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:22.556+0000 7f36917d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33115) UID: 0 2024-09-13T12:02:22.642 INFO:teuthology.orchestra.run.smithi005.stdout:osd.15: {} 2024-09-13T12:02:22.642 INFO:teuthology.orchestra.run.smithi005.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T12:02:22.646 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:22.646+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:22.693 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:22.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T12:02:22.747 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:22.746+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:22.848 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:22.848+0000 7f640e91b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33160) UID: 0 2024-09-13T12:02:22.849 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:22.849 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:22.857 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T12:02:22.949 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:22.948+0000 7f1ada246640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33458) UID: 0 2024-09-13T12:02:22.962 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:22.962 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:22.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T12:02:23.051 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:23.050+0000 7f4dbe8f8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33735) UID: 0 2024-09-13T12:02:23.073 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.074 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.082 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T12:02:23.152 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:23.151+0000 7f375a9c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33743) UID: 0 2024-09-13T12:02:23.185 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.185 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.194 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T12:02:23.253 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:23.253+0000 7f3235f26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33158) UID: 0 2024-09-13T12:02:23.296 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.297 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T12:02:23.354 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:23.353+0000 7fa064d67640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33113) UID: 0 2024-09-13T12:02:23.409 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.409 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T12:02:23.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T12:02:23.455 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:23.454+0000 7f933c49b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33155) UID: 0 2024-09-13T12:02:23.464 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:23.463+0000 7fa057764640 -1 osd.7 40 handle_osd_map injecting map crc failure 2024-09-13T12:02:23.465 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 min_write_recency_for_promote to 2 2024-09-13T12:02:23.481 INFO:teuthology.run_tasks:Running task rados... 2024-09-13T12:02:23.490 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.rados' 2024-09-13T12:02:23.723 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c7986b92c3a04dadb7cdbc40388c9cfa Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.rados' 2024-09-13T12:02:23.724 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T12:02:23.733 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T12:02:23.745 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.745 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T12:02:23.855 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.856 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T12:02:23.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T12:02:23.966 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:23.966 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:23.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T12:02:24.075 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a: {} 2024-09-13T12:02:24.075 INFO:teuthology.orchestra.run.smithi005.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:24.078 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:24.078 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:24.085 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b: {} 2024-09-13T12:02:24.085 INFO:teuthology.orchestra.run.smithi005.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:24.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T12:02:24.094 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c: {} 2024-09-13T12:02:24.095 INFO:teuthology.orchestra.run.smithi005.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T12:02:24.105 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T12:02:24.189 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:24.190 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:24.191 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:24.191+0000 7fbe39fa0640 -1 osd.2 40 handle_osd_map injecting map crc failure 2024-09-13T12:02:24.198 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T12:02:24.299 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:24.300 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:24.308 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T12:02:24.410 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-09-13T12:02:24.410 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T12:02:24.466 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is set 2024-09-13T12:02:24.466 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T12:02:24.467 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:24.465+0000 7ff4e7f97640 -1 osd.9 40 handle_osd_map injecting map crc failure 2024-09-13T12:02:24.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T12:02:24.803 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:24.802+0000 7f4dca859640 -1 osd.15 40 handle_osd_map injecting map crc failure 2024-09-13T12:02:24.889 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:24.888+0000 7f4db32f9640 -1 osd.0 41 handle_osd_map injecting map crc failure 2024-09-13T12:02:25.468 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T12:02:25.485 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T12:02:26.201 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:26.200+0000 7ff4e5f93640 -1 osd.9 42 handle_osd_map injecting map crc failure 2024-09-13T12:02:26.475 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T12:02:26.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T12:02:26.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-13T12:02:26.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:26.844+0000 7f568483e640 0 --2- 172.21.15.5:0/3807126433 >> v2:172.21.15.5:3300/0 conn(0x7f56801502f0 0x7f5680072a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T12:02:27.005 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:27.004+0000 7f83d6360640 -1 osd.3 43 handle_osd_map injecting map crc failure 2024-09-13T12:02:27.012 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:27.011+0000 7f5b85735640 -1 osd.8 43 handle_osd_map injecting map crc failure 2024-09-13T12:02:27.488 INFO:teuthology.orchestra.run.smithi005.stderr:noscrub is unset 2024-09-13T12:02:27.489 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T12:02:27.504 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T12:02:27.506 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T12:02:27.510 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:27.509+0000 7f36821c9640 -1 osd.11 43 handle_osd_map injecting map crc failure 2024-09-13T12:02:27.877 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:27.876+0000 7f83d6360640 -1 osd.3 44 handle_osd_map injecting map crc failure 2024-09-13T12:02:27.878 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:27.876+0000 7fbe39fa0640 -1 osd.2 44 handle_osd_map injecting map crc failure 2024-09-13T12:02:28.493 INFO:teuthology.orchestra.run.smithi005.stderr:nodeep-scrub is unset 2024-09-13T12:02:28.494 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T12:02:28.511 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-13T12:02:28.690 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:28.689+0000 7f4dca859640 -1 osd.15 45 handle_osd_map injecting map crc failure 2024-09-13T12:02:29.495 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-13T12:02:29.512 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T12:02:29.747 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:29.746+0000 7f5b85735640 -1 osd.8 46 handle_osd_map injecting map crc failure 2024-09-13T12:02:30.502 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T12:02:30.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-13T12:02:31.042 INFO:tasks.ceph.osd.10.smithi039.stderr:2024-09-13T12:02:31.040+0000 7f322a927640 -1 osd.10 47 handle_osd_map injecting map crc failure 2024-09-13T12:02:31.512 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-13T12:02:31.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T12:02:32.519 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T12:02:32.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-13T12:02:32.993 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:32.991+0000 7fbe39fa0640 -1 osd.2 48 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.454 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:33.452+0000 7f6401318640 -1 osd.14 48 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.527 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T12:02:33.532 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:33.531+0000 7fa057764640 -1 osd.7 48 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T12:02:33.569 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:33.568+0000 7f83d6360640 -1 osd.3 49 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.569 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:33.568+0000 7f328fcee640 -1 osd.13 48 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.942 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:33.941+0000 7f36821c9640 -1 osd.11 50 handle_osd_map injecting map crc failure 2024-09-13T12:02:33.943 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:02:33.941+0000 7f374f3c5640 -1 osd.12 50 handle_osd_map injecting map crc failure 2024-09-13T12:02:34.534 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T12:02:34.537 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:34.536+0000 7ff4e7f97640 -1 osd.9 51 handle_osd_map injecting map crc failure 2024-09-13T12:02:34.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T12:02:34.662 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:34.661+0000 7fa055760640 -1 osd.7 51 handle_osd_map injecting map crc failure 2024-09-13T12:02:35.546 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T12:02:35.566 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T12:02:36.564 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T12:02:36.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-13T12:02:37.349 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:37.347+0000 7f8c90162640 -1 osd.5 53 handle_osd_map injecting map crc failure 2024-09-13T12:02:37.565 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-13T12:02:37.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T12:02:38.577 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T12:02:38.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-13T12:02:39.179 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:39.178+0000 7f1acac3f640 -1 osd.1 55 handle_osd_map injecting map crc failure 2024-09-13T12:02:39.590 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-13T12:02:39.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T12:02:39.630 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:39.629+0000 7f36841cd640 -1 osd.11 55 handle_osd_map injecting map crc failure 2024-09-13T12:02:40.563 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:40.561+0000 7fa055760640 -1 osd.7 55 handle_osd_map injecting map crc failure 2024-09-13T12:02:40.601 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T12:02:40.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T12:02:40.750 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:40.748+0000 7f932ce94640 -1 osd.6 55 handle_osd_map injecting map crc failure 2024-09-13T12:02:41.292 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:41.291+0000 7f83d6360640 -1 osd.3 57 handle_osd_map injecting map crc failure 2024-09-13T12:02:41.609 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T12:02:41.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T12:02:41.792 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:41.790+0000 7fbe39fa0640 -1 osd.2 58 handle_osd_map injecting map crc failure 2024-09-13T12:02:41.793 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:41.791+0000 7f1acac3f640 -1 osd.1 58 handle_osd_map injecting map crc failure 2024-09-13T12:02:42.112 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:42.111+0000 7ff4e5f93640 -1 osd.9 58 handle_osd_map injecting map crc failure 2024-09-13T12:02:42.619 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:02:42.618+0000 7f8c92166640 -1 osd.5 59 handle_osd_map injecting map crc failure 2024-09-13T12:02:42.620 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T12:02:42.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-13T12:02:43.626 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T12:02:43.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T12:02:44.514 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:02:44.513+0000 7f4db32f9640 -1 osd.0 60 handle_osd_map injecting map crc failure 2024-09-13T12:02:44.639 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T12:02:44.639 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:02:44.638+0000 7f5b87739640 -1 osd.8 61 handle_osd_map injecting map crc failure 2024-09-13T12:02:44.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-13T12:02:44.981 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:02:44.979+0000 7f1acac3f640 -1 osd.1 60 handle_osd_map injecting map crc failure 2024-09-13T12:02:45.062 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:02:45.060+0000 7f63ff314640 -1 osd.14 60 handle_osd_map injecting map crc failure 2024-09-13T12:02:45.651 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-13T12:02:45.668 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T12:02:45.704 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:45.703+0000 7fc765526640 -1 osd.4 62 handle_osd_map injecting map crc failure 2024-09-13T12:02:46.657 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T12:02:46.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-13T12:02:47.443 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:47.442+0000 7f932ce94640 -1 osd.6 63 handle_osd_map injecting map crc failure 2024-09-13T12:02:47.664 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-13T12:02:47.683 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T12:02:48.672 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T12:02:48.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-13T12:02:49.205 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:49.203+0000 7fc765526640 -1 osd.4 64 handle_osd_map injecting map crc failure 2024-09-13T12:02:49.539 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:49.537+0000 7fa057764640 -1 osd.7 64 handle_osd_map injecting map crc failure 2024-09-13T12:02:49.678 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T12:02:49.692 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:02:49.691+0000 7f932ee98640 -1 osd.6 64 handle_osd_map injecting map crc failure 2024-09-13T12:02:49.696 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T12:02:49.699 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:49.698+0000 7f83d6360640 -1 osd.3 64 handle_osd_map injecting map crc failure 2024-09-13T12:02:49.815 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:02:49.814+0000 7f4dca859640 -1 osd.15 66 handle_osd_map injecting map crc failure 2024-09-13T12:02:50.695 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T12:02:50.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-13T12:02:51.373 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:51.371+0000 7fc765526640 -1 osd.4 67 handle_osd_map injecting map crc failure 2024-09-13T12:02:51.374 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:51.373+0000 7f328fcee640 -1 osd.13 67 handle_osd_map injecting map crc failure 2024-09-13T12:02:51.375 INFO:tasks.ceph.osd.3.smithi087.stderr:2024-09-13T12:02:51.372+0000 7f83d6360640 -1 osd.3 67 handle_osd_map injecting map crc failure 2024-09-13T12:02:51.705 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T12:02:51.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T12:02:52.713 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T12:02:52.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-13T12:02:52.744 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:02:52.743+0000 7f328fcee640 -1 osd.13 69 handle_osd_map injecting map crc failure 2024-09-13T12:02:53.729 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-13T12:02:53.732 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:02:53.728+0000 7ff4e7f97640 -1 osd.9 70 handle_osd_map injecting map crc failure 2024-09-13T12:02:53.746 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T12:02:53.863 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:53.861+0000 7fc765526640 -1 osd.4 69 handle_osd_map injecting map crc failure 2024-09-13T12:02:53.864 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:02:53.862+0000 7f36821c9640 -1 osd.11 70 handle_osd_map injecting map crc failure 2024-09-13T12:02:54.566 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:02:54.564+0000 7fbe39fa0640 -1 osd.2 69 handle_osd_map injecting map crc failure 2024-09-13T12:02:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T12:02:54.731 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:02:54.728+0000 7fc76752a640 -1 osd.4 71 handle_osd_map injecting map crc failure 2024-09-13T12:02:54.748 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-13T12:02:56.001 INFO:teuthology.orchestra.run.smithi005.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-13T12:02:56.018 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T12:02:56.019 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T12:02:56.019 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T12:02:56.019 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T12:02:56.019 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T12:02:56.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:02:56.196 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:02:56.195+0000 7fa055760640 -1 osd.7 71 handle_osd_map injecting map crc failure 2024-09-13T12:02:56.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-09-13T12:02:56.330+0000 7f02d23ce640 0 --2- 172.21.15.5:0/2788460740 >> v2:172.21.15.5:3300/0 conn(0x7f02cc155f20 0x7f02cc15e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-13T12:02:56.533 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:56.533 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":73,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:02:55.999714+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T12:01:59.896737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-13T12:02:11.639758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-13T12:02:13.729433+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":2,"min_write_recency_for_promote":2,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8.0600004196166992,"score_stable":8.0600004196166992,"optimal_score":0.31000000238418579,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:02:56.546 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T12:02:56.546 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:02:56.847 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:56.847 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":73,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:02:55.999714+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T12:01:59.896737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-13T12:02:11.639758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-13T12:02:13.729433+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":2,"min_write_recency_for_promote":2,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8.0600004196166992,"score_stable":8.0600004196166992,"optimal_score":0.31000000238418579,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:02:56.860 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T12:02:56.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T12:02:56.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T12:02:56.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T12:02:56.862 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T12:02:56.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T12:02:56.863 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T12:02:56.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T12:02:56.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T12:02:56.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T12:02:56.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T12:02:56.865 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T12:02:56.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T12:02:56.866 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T12:02:56.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T12:02:56.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T12:02:57.284 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:57.285 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T12:02:57.327 INFO:teuthology.orchestra.run.smithi005.stdout:94489280527 2024-09-13T12:02:57.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T12:02:57.329 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:57.329 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T12:02:57.346 INFO:teuthology.orchestra.run.smithi005.stdout:94489280528 2024-09-13T12:02:57.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T12:02:57.350 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:57.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T12:02:57.364 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:57.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T12:02:57.367 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:57.367 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T12:02:57.368 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:57.368 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T12:02:57.375 INFO:teuthology.orchestra.run.smithi005.stdout:98784247822 2024-09-13T12:02:57.375 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T12:02:57.385 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:57.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T12:02:57.390 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:57.391 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:57.391 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T12:02:57.392 INFO:teuthology.orchestra.run.smithi005.stdout:94489280526 2024-09-13T12:02:57.392 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T12:02:57.393 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T12:02:57.401 INFO:teuthology.orchestra.run.smithi005.stdout:90194313232 2024-09-13T12:02:57.402 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T12:02:57.410 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:57.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T12:02:57.434 INFO:teuthology.orchestra.run.smithi005.stdout:94489280527 2024-09-13T12:02:57.434 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T12:02:58.383 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:58.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247824 got 98784247824 for osd.7 2024-09-13T12:02:58.416 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.538 INFO:teuthology.orchestra.run.smithi005.stdout:94489280527 2024-09-13T12:02:58.550 INFO:teuthology.orchestra.run.smithi005.stdout:94489280528 2024-09-13T12:02:58.559 INFO:teuthology.orchestra.run.smithi005.stdout:98784247822 2024-09-13T12:02:58.560 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:58.563 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:58.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280527 got 94489280527 for osd.4 2024-09-13T12:02:58.566 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247822 got 98784247822 for osd.15 2024-09-13T12:02:58.584 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247824 got 98784247824 for osd.3 2024-09-13T12:02:58.598 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247824 got 98784247824 for osd.9 2024-09-13T12:02:58.600 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:58.600 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280528 got 94489280528 for osd.8 2024-09-13T12:02:58.607 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.625 INFO:teuthology.orchestra.run.smithi005.stdout:98784247824 2024-09-13T12:02:58.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247825 got 98784247825 for osd.1 2024-09-13T12:02:58.627 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.630 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:58.630 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:58.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247824 got 98784247824 for osd.13 2024-09-13T12:02:58.644 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.646 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:58.647 INFO:teuthology.orchestra.run.smithi005.stdout:98784247823 2024-09-13T12:02:58.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247823 got 98784247823 for osd.5 2024-09-13T12:02:58.653 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247823 got 98784247823 for osd.14 2024-09-13T12:02:58.658 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.661 INFO:teuthology.orchestra.run.smithi005.stdout:90194313232 2024-09-13T12:02:58.666 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247825 got 98784247825 for osd.6 2024-09-13T12:02:58.666 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247823 got 98784247823 for osd.11 2024-09-13T12:02:58.668 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.669 INFO:teuthology.orchestra.run.smithi005.stdout:98784247825 2024-09-13T12:02:58.674 INFO:teuthology.orchestra.run.smithi005.stdout:94489280526 2024-09-13T12:02:58.676 INFO:teuthology.orchestra.run.smithi005.stdout:94489280527 2024-09-13T12:02:58.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313232 got 90194313232 for osd.12 2024-09-13T12:02:58.679 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247825 got 98784247825 for osd.2 2024-09-13T12:02:58.687 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280526 got 94489280526 for osd.10 2024-09-13T12:02:58.691 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280527 got 94489280527 for osd.0 2024-09-13T12:02:58.693 DEBUG:teuthology.parallel:result is None 2024-09-13T12:02:58.693 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T12:02:58.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:59.034 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:59.035 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:59.047 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-13T12:02:57.572961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455748,"kb_used_data":10672,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493692,"statfs":{"total":1546188226560,"available":1545721540608,"internally_reserved":0,"allocated":10928128,"data_stored":7499852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.854822"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":240,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047700+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:56.047700+0000","last_peered":"2024-09-13T12:02:56.047700+0000","last_clean":"2024-09-13T12:02:56.047700+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:56.047700+0000","last_undegraded":"2024-09-13T12:02:56.047700+0000","last_fullsized":"2024-09-13T12:02:56.047700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006507+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:56.006507+0000","last_peered":"2024-09-13T12:02:56.006507+0000","last_clean":"2024-09-13T12:02:56.006507+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:56.006507+0000","last_undegraded":"2024-09-13T12:02:56.006507+0000","last_fullsized":"2024-09-13T12:02:56.006507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006501+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:02:56.006501+0000","last_peered":"2024-09-13T12:02:56.006501+0000","last_clean":"2024-09-13T12:02:56.006501+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:02:56.006501+0000","last_undegraded":"2024-09-13T12:02:56.006501+0000","last_fullsized":"2024-09-13T12:02:56.006501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202485+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:56.202485+0000","last_peered":"2024-09-13T12:02:56.202485+0000","last_clean":"2024-09-13T12:02:56.202485+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:56.202485+0000","last_undegraded":"2024-09-13T12:02:56.202485+0000","last_fullsized":"2024-09-13T12:02:56.202485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:13.385728+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.589502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006461+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:56.006461+0000","last_peered":"2024-09-13T12:02:56.006461+0000","last_clean":"2024-09-13T12:02:56.006461+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:56.006461+0000","last_undegraded":"2024-09-13T12:02:56.006461+0000","last_fullsized":"2024-09-13T12:02:56.006461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004398+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:02:56.004398+0000","last_peered":"2024-09-13T12:02:56.004398+0000","last_clean":"2024-09-13T12:02:56.004398+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:02:56.004398+0000","last_undegraded":"2024-09-13T12:02:56.004398+0000","last_fullsized":"2024-09-13T12:02:56.004398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":106,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198956+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:56.198956+0000","last_peered":"2024-09-13T12:02:56.198956+0000","last_clean":"2024-09-13T12:02:56.198956+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:56.198956+0000","last_undegraded":"2024-09-13T12:02:56.198956+0000","last_fullsized":"2024-09-13T12:02:56.198956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.479180+0000","last_change":"2024-09-13T12:02:13.470037+0000","last_active":"2024-09-13T12:02:56.479180+0000","last_peered":"2024-09-13T12:02:56.479180+0000","last_clean":"2024-09-13T12:02:56.479180+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:02:56.479180+0000","last_undegraded":"2024-09-13T12:02:56.479180+0000","last_fullsized":"2024-09-13T12:02:56.479180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:39.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047718+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:02:56.047718+0000","last_peered":"2024-09-13T12:02:56.047718+0000","last_clean":"2024-09-13T12:02:56.047718+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:02:56.047718+0000","last_undegraded":"2024-09-13T12:02:56.047718+0000","last_fullsized":"2024-09-13T12:02:56.047718+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197032+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:56.197032+0000","last_peered":"2024-09-13T12:02:56.197032+0000","last_clean":"2024-09-13T12:02:56.197032+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:56.197032+0000","last_undegraded":"2024-09-13T12:02:56.197032+0000","last_fullsized":"2024-09-13T12:02:56.197032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199635+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:02:56.199635+0000","last_peered":"2024-09-13T12:02:56.199635+0000","last_clean":"2024-09-13T12:02:56.199635+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:02:56.199635+0000","last_undegraded":"2024-09-13T12:02:56.199635+0000","last_fullsized":"2024-09-13T12:02:56.199635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202968+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:56.202968+0000","last_peered":"2024-09-13T12:02:56.202968+0000","last_clean":"2024-09-13T12:02:56.202968+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:56.202968+0000","last_undegraded":"2024-09-13T12:02:56.202968+0000","last_fullsized":"2024-09-13T12:02:56.202968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":97,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:15.394122+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:30.314256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004305+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:56.004305+0000","last_peered":"2024-09-13T12:02:56.004305+0000","last_clean":"2024-09-13T12:02:56.004305+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:56.004305+0000","last_undegraded":"2024-09-13T12:02:56.004305+0000","last_fullsized":"2024-09-13T12:02:56.004305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199634+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:02:56.199634+0000","last_peered":"2024-09-13T12:02:56.199634+0000","last_clean":"2024-09-13T12:02:56.199634+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:02:56.199634+0000","last_undegraded":"2024-09-13T12:02:56.199634+0000","last_fullsized":"2024-09-13T12:02:56.199634+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198943+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:56.198943+0000","last_peered":"2024-09-13T12:02:56.198943+0000","last_clean":"2024-09-13T12:02:56.198943+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:56.198943+0000","last_undegraded":"2024-09-13T12:02:56.198943+0000","last_fullsized":"2024-09-13T12:02:56.198943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":400594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":397088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":397950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280527,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":400245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":401437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":395957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":991447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247824,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":401357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247823,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":399813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:59.049 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:59.345 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:59.345 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:59.357 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-13T12:02:57.572961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455748,"kb_used_data":10672,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493692,"statfs":{"total":1546188226560,"available":1545721540608,"internally_reserved":0,"allocated":10928128,"data_stored":7499852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.854822"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":240,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047700+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:56.047700+0000","last_peered":"2024-09-13T12:02:56.047700+0000","last_clean":"2024-09-13T12:02:56.047700+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:56.047700+0000","last_undegraded":"2024-09-13T12:02:56.047700+0000","last_fullsized":"2024-09-13T12:02:56.047700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006507+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:56.006507+0000","last_peered":"2024-09-13T12:02:56.006507+0000","last_clean":"2024-09-13T12:02:56.006507+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:56.006507+0000","last_undegraded":"2024-09-13T12:02:56.006507+0000","last_fullsized":"2024-09-13T12:02:56.006507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006501+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:02:56.006501+0000","last_peered":"2024-09-13T12:02:56.006501+0000","last_clean":"2024-09-13T12:02:56.006501+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:02:56.006501+0000","last_undegraded":"2024-09-13T12:02:56.006501+0000","last_fullsized":"2024-09-13T12:02:56.006501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202485+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:56.202485+0000","last_peered":"2024-09-13T12:02:56.202485+0000","last_clean":"2024-09-13T12:02:56.202485+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:56.202485+0000","last_undegraded":"2024-09-13T12:02:56.202485+0000","last_fullsized":"2024-09-13T12:02:56.202485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:13.385728+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.589502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006461+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:56.006461+0000","last_peered":"2024-09-13T12:02:56.006461+0000","last_clean":"2024-09-13T12:02:56.006461+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:56.006461+0000","last_undegraded":"2024-09-13T12:02:56.006461+0000","last_fullsized":"2024-09-13T12:02:56.006461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004398+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:02:56.004398+0000","last_peered":"2024-09-13T12:02:56.004398+0000","last_clean":"2024-09-13T12:02:56.004398+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:02:56.004398+0000","last_undegraded":"2024-09-13T12:02:56.004398+0000","last_fullsized":"2024-09-13T12:02:56.004398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":106,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198956+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:56.198956+0000","last_peered":"2024-09-13T12:02:56.198956+0000","last_clean":"2024-09-13T12:02:56.198956+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:56.198956+0000","last_undegraded":"2024-09-13T12:02:56.198956+0000","last_fullsized":"2024-09-13T12:02:56.198956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.479180+0000","last_change":"2024-09-13T12:02:13.470037+0000","last_active":"2024-09-13T12:02:56.479180+0000","last_peered":"2024-09-13T12:02:56.479180+0000","last_clean":"2024-09-13T12:02:56.479180+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:02:56.479180+0000","last_undegraded":"2024-09-13T12:02:56.479180+0000","last_fullsized":"2024-09-13T12:02:56.479180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:39.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047718+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:02:56.047718+0000","last_peered":"2024-09-13T12:02:56.047718+0000","last_clean":"2024-09-13T12:02:56.047718+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:02:56.047718+0000","last_undegraded":"2024-09-13T12:02:56.047718+0000","last_fullsized":"2024-09-13T12:02:56.047718+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197032+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:56.197032+0000","last_peered":"2024-09-13T12:02:56.197032+0000","last_clean":"2024-09-13T12:02:56.197032+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:56.197032+0000","last_undegraded":"2024-09-13T12:02:56.197032+0000","last_fullsized":"2024-09-13T12:02:56.197032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199635+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:02:56.199635+0000","last_peered":"2024-09-13T12:02:56.199635+0000","last_clean":"2024-09-13T12:02:56.199635+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:02:56.199635+0000","last_undegraded":"2024-09-13T12:02:56.199635+0000","last_fullsized":"2024-09-13T12:02:56.199635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202968+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:56.202968+0000","last_peered":"2024-09-13T12:02:56.202968+0000","last_clean":"2024-09-13T12:02:56.202968+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:56.202968+0000","last_undegraded":"2024-09-13T12:02:56.202968+0000","last_fullsized":"2024-09-13T12:02:56.202968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":97,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:15.394122+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:30.314256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004305+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:56.004305+0000","last_peered":"2024-09-13T12:02:56.004305+0000","last_clean":"2024-09-13T12:02:56.004305+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:56.004305+0000","last_undegraded":"2024-09-13T12:02:56.004305+0000","last_fullsized":"2024-09-13T12:02:56.004305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199634+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:02:56.199634+0000","last_peered":"2024-09-13T12:02:56.199634+0000","last_clean":"2024-09-13T12:02:56.199634+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:02:56.199634+0000","last_undegraded":"2024-09-13T12:02:56.199634+0000","last_fullsized":"2024-09-13T12:02:56.199634+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198943+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:56.198943+0000","last_peered":"2024-09-13T12:02:56.198943+0000","last_clean":"2024-09-13T12:02:56.198943+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:56.198943+0000","last_undegraded":"2024-09-13T12:02:56.198943+0000","last_fullsized":"2024-09-13T12:02:56.198943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":400594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":397088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":397950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280527,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":400245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":401437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247825,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":395957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":991447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247824,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":401357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247823,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":399813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:02:59.358 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T12:02:59.359 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T12:02:59.677 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T12:02:59.686 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T12:02:59.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:02:59.986 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:02:59.987 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:02:59.999 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-13T12:02:59.573556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455748,"kb_used_data":10672,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493692,"statfs":{"total":1546188226560,"available":1545721540608,"internally_reserved":0,"allocated":10928128,"data_stored":7499852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001437"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":240,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047700+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:56.047700+0000","last_peered":"2024-09-13T12:02:56.047700+0000","last_clean":"2024-09-13T12:02:56.047700+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:56.047700+0000","last_undegraded":"2024-09-13T12:02:56.047700+0000","last_fullsized":"2024-09-13T12:02:56.047700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006507+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:56.006507+0000","last_peered":"2024-09-13T12:02:56.006507+0000","last_clean":"2024-09-13T12:02:56.006507+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:56.006507+0000","last_undegraded":"2024-09-13T12:02:56.006507+0000","last_fullsized":"2024-09-13T12:02:56.006507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006501+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:02:56.006501+0000","last_peered":"2024-09-13T12:02:56.006501+0000","last_clean":"2024-09-13T12:02:56.006501+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:02:56.006501+0000","last_undegraded":"2024-09-13T12:02:56.006501+0000","last_fullsized":"2024-09-13T12:02:56.006501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202485+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:56.202485+0000","last_peered":"2024-09-13T12:02:56.202485+0000","last_clean":"2024-09-13T12:02:56.202485+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:56.202485+0000","last_undegraded":"2024-09-13T12:02:56.202485+0000","last_fullsized":"2024-09-13T12:02:56.202485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:13.385728+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.589502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006461+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:56.006461+0000","last_peered":"2024-09-13T12:02:56.006461+0000","last_clean":"2024-09-13T12:02:56.006461+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:56.006461+0000","last_undegraded":"2024-09-13T12:02:56.006461+0000","last_fullsized":"2024-09-13T12:02:56.006461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004398+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:02:56.004398+0000","last_peered":"2024-09-13T12:02:56.004398+0000","last_clean":"2024-09-13T12:02:56.004398+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:02:56.004398+0000","last_undegraded":"2024-09-13T12:02:56.004398+0000","last_fullsized":"2024-09-13T12:02:56.004398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":106,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198956+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:56.198956+0000","last_peered":"2024-09-13T12:02:56.198956+0000","last_clean":"2024-09-13T12:02:56.198956+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:56.198956+0000","last_undegraded":"2024-09-13T12:02:56.198956+0000","last_fullsized":"2024-09-13T12:02:56.198956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.479180+0000","last_change":"2024-09-13T12:02:13.470037+0000","last_active":"2024-09-13T12:02:56.479180+0000","last_peered":"2024-09-13T12:02:56.479180+0000","last_clean":"2024-09-13T12:02:56.479180+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:02:56.479180+0000","last_undegraded":"2024-09-13T12:02:56.479180+0000","last_fullsized":"2024-09-13T12:02:56.479180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:39.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047718+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:02:56.047718+0000","last_peered":"2024-09-13T12:02:56.047718+0000","last_clean":"2024-09-13T12:02:56.047718+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:02:56.047718+0000","last_undegraded":"2024-09-13T12:02:56.047718+0000","last_fullsized":"2024-09-13T12:02:56.047718+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197032+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:56.197032+0000","last_peered":"2024-09-13T12:02:56.197032+0000","last_clean":"2024-09-13T12:02:56.197032+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:56.197032+0000","last_undegraded":"2024-09-13T12:02:56.197032+0000","last_fullsized":"2024-09-13T12:02:56.197032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199635+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:02:56.199635+0000","last_peered":"2024-09-13T12:02:56.199635+0000","last_clean":"2024-09-13T12:02:56.199635+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:02:56.199635+0000","last_undegraded":"2024-09-13T12:02:56.199635+0000","last_fullsized":"2024-09-13T12:02:56.199635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202968+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:56.202968+0000","last_peered":"2024-09-13T12:02:56.202968+0000","last_clean":"2024-09-13T12:02:56.202968+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:56.202968+0000","last_undegraded":"2024-09-13T12:02:56.202968+0000","last_fullsized":"2024-09-13T12:02:56.202968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":97,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:15.394122+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:30.314256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004305+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:56.004305+0000","last_peered":"2024-09-13T12:02:56.004305+0000","last_clean":"2024-09-13T12:02:56.004305+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:56.004305+0000","last_undegraded":"2024-09-13T12:02:56.004305+0000","last_fullsized":"2024-09-13T12:02:56.004305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199634+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:02:56.199634+0000","last_peered":"2024-09-13T12:02:56.199634+0000","last_clean":"2024-09-13T12:02:56.199634+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:02:56.199634+0000","last_undegraded":"2024-09-13T12:02:56.199634+0000","last_fullsized":"2024-09-13T12:02:56.199634+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198943+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:56.198943+0000","last_peered":"2024-09-13T12:02:56.198943+0000","last_clean":"2024-09-13T12:02:56.198943+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:56.198943+0000","last_undegraded":"2024-09-13T12:02:56.198943+0000","last_fullsized":"2024-09-13T12:02:56.198943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":400594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":397088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":397950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":400245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":401437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":395957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":991447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":401357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247824,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":399813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:03:00.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:03:00.298 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:03:00.299 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:03:00.311 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-13T12:02:59.573556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455748,"kb_used_data":10672,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493692,"statfs":{"total":1546188226560,"available":1545721540608,"internally_reserved":0,"allocated":10928128,"data_stored":7499852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001437"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":240,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047700+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:56.047700+0000","last_peered":"2024-09-13T12:02:56.047700+0000","last_clean":"2024-09-13T12:02:56.047700+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:56.047700+0000","last_undegraded":"2024-09-13T12:02:56.047700+0000","last_fullsized":"2024-09-13T12:02:56.047700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006507+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:56.006507+0000","last_peered":"2024-09-13T12:02:56.006507+0000","last_clean":"2024-09-13T12:02:56.006507+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:56.006507+0000","last_undegraded":"2024-09-13T12:02:56.006507+0000","last_fullsized":"2024-09-13T12:02:56.006507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006501+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:02:56.006501+0000","last_peered":"2024-09-13T12:02:56.006501+0000","last_clean":"2024-09-13T12:02:56.006501+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:02:56.006501+0000","last_undegraded":"2024-09-13T12:02:56.006501+0000","last_fullsized":"2024-09-13T12:02:56.006501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202485+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:56.202485+0000","last_peered":"2024-09-13T12:02:56.202485+0000","last_clean":"2024-09-13T12:02:56.202485+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:56.202485+0000","last_undegraded":"2024-09-13T12:02:56.202485+0000","last_fullsized":"2024-09-13T12:02:56.202485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:13.385728+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.589502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006461+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:56.006461+0000","last_peered":"2024-09-13T12:02:56.006461+0000","last_clean":"2024-09-13T12:02:56.006461+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:56.006461+0000","last_undegraded":"2024-09-13T12:02:56.006461+0000","last_fullsized":"2024-09-13T12:02:56.006461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004398+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:02:56.004398+0000","last_peered":"2024-09-13T12:02:56.004398+0000","last_clean":"2024-09-13T12:02:56.004398+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:02:56.004398+0000","last_undegraded":"2024-09-13T12:02:56.004398+0000","last_fullsized":"2024-09-13T12:02:56.004398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":106,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198956+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:56.198956+0000","last_peered":"2024-09-13T12:02:56.198956+0000","last_clean":"2024-09-13T12:02:56.198956+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:56.198956+0000","last_undegraded":"2024-09-13T12:02:56.198956+0000","last_fullsized":"2024-09-13T12:02:56.198956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.479180+0000","last_change":"2024-09-13T12:02:13.470037+0000","last_active":"2024-09-13T12:02:56.479180+0000","last_peered":"2024-09-13T12:02:56.479180+0000","last_clean":"2024-09-13T12:02:56.479180+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:02:56.479180+0000","last_undegraded":"2024-09-13T12:02:56.479180+0000","last_fullsized":"2024-09-13T12:02:56.479180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:39.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047718+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:02:56.047718+0000","last_peered":"2024-09-13T12:02:56.047718+0000","last_clean":"2024-09-13T12:02:56.047718+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:02:56.047718+0000","last_undegraded":"2024-09-13T12:02:56.047718+0000","last_fullsized":"2024-09-13T12:02:56.047718+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197032+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:56.197032+0000","last_peered":"2024-09-13T12:02:56.197032+0000","last_clean":"2024-09-13T12:02:56.197032+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:56.197032+0000","last_undegraded":"2024-09-13T12:02:56.197032+0000","last_fullsized":"2024-09-13T12:02:56.197032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199635+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:02:56.199635+0000","last_peered":"2024-09-13T12:02:56.199635+0000","last_clean":"2024-09-13T12:02:56.199635+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:02:56.199635+0000","last_undegraded":"2024-09-13T12:02:56.199635+0000","last_fullsized":"2024-09-13T12:02:56.199635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202968+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:56.202968+0000","last_peered":"2024-09-13T12:02:56.202968+0000","last_clean":"2024-09-13T12:02:56.202968+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:56.202968+0000","last_undegraded":"2024-09-13T12:02:56.202968+0000","last_fullsized":"2024-09-13T12:02:56.202968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":97,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:15.394122+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:30.314256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004305+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:56.004305+0000","last_peered":"2024-09-13T12:02:56.004305+0000","last_clean":"2024-09-13T12:02:56.004305+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:56.004305+0000","last_undegraded":"2024-09-13T12:02:56.004305+0000","last_fullsized":"2024-09-13T12:02:56.004305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199634+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:02:56.199634+0000","last_peered":"2024-09-13T12:02:56.199634+0000","last_clean":"2024-09-13T12:02:56.199634+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:02:56.199634+0000","last_undegraded":"2024-09-13T12:02:56.199634+0000","last_fullsized":"2024-09-13T12:02:56.199634+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198943+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:56.198943+0000","last_peered":"2024-09-13T12:02:56.198943+0000","last_clean":"2024-09-13T12:02:56.198943+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:56.198943+0000","last_undegraded":"2024-09-13T12:02:56.198943+0000","last_fullsized":"2024-09-13T12:02:56.198943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":400594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":397088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":397950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":400245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":401437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":395957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":991447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":401357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247824,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":399813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:03:00.312 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T12:03:00.312 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:03:00.610 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:03:00.611 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:03:00.623 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-13T12:02:59.573556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455748,"kb_used_data":10672,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493692,"statfs":{"total":1546188226560,"available":1545721540608,"internally_reserved":0,"allocated":10928128,"data_stored":7499852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001437"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":240,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047700+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:02:56.047700+0000","last_peered":"2024-09-13T12:02:56.047700+0000","last_clean":"2024-09-13T12:02:56.047700+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:02:56.047700+0000","last_undegraded":"2024-09-13T12:02:56.047700+0000","last_fullsized":"2024-09-13T12:02:56.047700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006507+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:02:56.006507+0000","last_peered":"2024-09-13T12:02:56.006507+0000","last_clean":"2024-09-13T12:02:56.006507+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:02:56.006507+0000","last_undegraded":"2024-09-13T12:02:56.006507+0000","last_fullsized":"2024-09-13T12:02:56.006507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006501+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:02:56.006501+0000","last_peered":"2024-09-13T12:02:56.006501+0000","last_clean":"2024-09-13T12:02:56.006501+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:02:56.006501+0000","last_undegraded":"2024-09-13T12:02:56.006501+0000","last_fullsized":"2024-09-13T12:02:56.006501+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202485+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:02:56.202485+0000","last_peered":"2024-09-13T12:02:56.202485+0000","last_clean":"2024-09-13T12:02:56.202485+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:02:56.202485+0000","last_undegraded":"2024-09-13T12:02:56.202485+0000","last_fullsized":"2024-09-13T12:02:56.202485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:13.385728+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.589502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.006461+0000","last_change":"2024-09-13T12:02:03.357586+0000","last_active":"2024-09-13T12:02:56.006461+0000","last_peered":"2024-09-13T12:02:56.006461+0000","last_clean":"2024-09-13T12:02:56.006461+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:02:56.006461+0000","last_undegraded":"2024-09-13T12:02:56.006461+0000","last_fullsized":"2024-09-13T12:02:56.006461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:02.038951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":94,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004398+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:02:56.004398+0000","last_peered":"2024-09-13T12:02:56.004398+0000","last_clean":"2024-09-13T12:02:56.004398+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:02:56.004398+0000","last_undegraded":"2024-09-13T12:02:56.004398+0000","last_fullsized":"2024-09-13T12:02:56.004398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":106,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198956+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:02:56.198956+0000","last_peered":"2024-09-13T12:02:56.198956+0000","last_clean":"2024-09-13T12:02:56.198956+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:02:56.198956+0000","last_undegraded":"2024-09-13T12:02:56.198956+0000","last_fullsized":"2024-09-13T12:02:56.198956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.479180+0000","last_change":"2024-09-13T12:02:13.470037+0000","last_active":"2024-09-13T12:02:56.479180+0000","last_peered":"2024-09-13T12:02:56.479180+0000","last_clean":"2024-09-13T12:02:56.479180+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:02:56.479180+0000","last_undegraded":"2024-09-13T12:02:56.479180+0000","last_fullsized":"2024-09-13T12:02:56.479180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:39.283879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.047718+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:02:56.047718+0000","last_peered":"2024-09-13T12:02:56.047718+0000","last_clean":"2024-09-13T12:02:56.047718+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:02:56.047718+0000","last_undegraded":"2024-09-13T12:02:56.047718+0000","last_fullsized":"2024-09-13T12:02:56.047718+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":107,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197032+0000","last_change":"2024-09-13T12:02:03.358024+0000","last_active":"2024-09-13T12:02:56.197032+0000","last_peered":"2024-09-13T12:02:56.197032+0000","last_clean":"2024-09-13T12:02:56.197032+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:02:56.197032+0000","last_undegraded":"2024-09-13T12:02:56.197032+0000","last_fullsized":"2024-09-13T12:02:56.197032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:10.477299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199635+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:02:56.199635+0000","last_peered":"2024-09-13T12:02:56.199635+0000","last_clean":"2024-09-13T12:02:56.199635+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:02:56.199635+0000","last_undegraded":"2024-09-13T12:02:56.199635+0000","last_fullsized":"2024-09-13T12:02:56.199635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.202968+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:02:56.202968+0000","last_peered":"2024-09-13T12:02:56.202968+0000","last_clean":"2024-09-13T12:02:56.202968+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:02:56.202968+0000","last_undegraded":"2024-09-13T12:02:56.202968+0000","last_fullsized":"2024-09-13T12:02:56.202968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":97,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.197022+0000","last_change":"2024-09-13T12:02:15.394122+0000","last_active":"2024-09-13T12:02:56.197022+0000","last_peered":"2024-09-13T12:02:56.197022+0000","last_clean":"2024-09-13T12:02:56.197022+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:02:56.197022+0000","last_undegraded":"2024-09-13T12:02:56.197022+0000","last_fullsized":"2024-09-13T12:02:56.197022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:30.314256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.004305+0000","last_change":"2024-09-13T12:02:03.323161+0000","last_active":"2024-09-13T12:02:56.004305+0000","last_peered":"2024-09-13T12:02:56.004305+0000","last_clean":"2024-09-13T12:02:56.004305+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:02:56.004305+0000","last_undegraded":"2024-09-13T12:02:56.004305+0000","last_fullsized":"2024-09-13T12:02:56.004305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:20.080520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.199634+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:02:56.199634+0000","last_peered":"2024-09-13T12:02:56.199634+0000","last_clean":"2024-09-13T12:02:56.199634+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:02:56.199634+0000","last_undegraded":"2024-09-13T12:02:56.199634+0000","last_fullsized":"2024-09-13T12:02:56.199634+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T12:02:56.198943+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:02:56.198943+0000","last_peered":"2024-09-13T12:02:56.198943+0000","last_clean":"2024-09-13T12:02:56.198943+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:02:56.198943+0000","last_undegraded":"2024-09-13T12:02:56.198943+0000","last_fullsized":"2024-09-13T12:02:56.198943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247823,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313232,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":400594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":397088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":397950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":614400,"data_stored":400245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":401437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247826,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":606208,"data_stored":397593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":606208,"data_stored":397951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":395957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":991447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":401357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247824,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":399813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:03:00.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T12:03:00.923 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:03:00.923 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":73,"fsid":"0694c701-d7eb-494e-ad86-64faa4809c32","created":"2024-09-13T12:01:45.016372+0000","modified":"2024-09-13T12:02:55.999714+0000","last_up_change":"2024-09-13T12:01:58.267157+0000","last_in_change":"2024-09-13T12:01:49.473230+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T12:01:57.589370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T12:01:59.896737+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-13T12:02:11.639758+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9800000190734863,"score_stable":3.9800000190734863,"optimal_score":0.43999999761581421,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-13T12:02:13.729433+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"35","last_force_op_resend_prenautilus":"35","last_force_op_resend_preluminous":"35","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":33,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":250,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":60,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":2,"min_write_recency_for_promote":2,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8.0600004196166992,"score_stable":8.0600004196166992,"optimal_score":0.31000000238418579,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17c551c2-db05-4b58-af49-885510ae31ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":3436814551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6805","nonce":3436814551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6807","nonce":3436814551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":3436814551}]},"public_addr":"172.21.15.5:6804/3436814551","cluster_addr":"172.21.15.5:6805/3436814551","heartbeat_back_addr":"172.21.15.5:6807/3436814551","heartbeat_front_addr":"172.21.15.5:6806/3436814551","state":["exists","up"]},{"osd":1,"uuid":"a19df369-47ed-4f8c-a422-f009d11c2360","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1772774403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1772774403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":1772774403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":1772774403}]},"public_addr":"172.21.15.37:6808/1772774403","cluster_addr":"172.21.15.37:6809/1772774403","heartbeat_back_addr":"172.21.15.37:6811/1772774403","heartbeat_front_addr":"172.21.15.37:6810/1772774403","state":["exists","up"]},{"osd":2,"uuid":"990d3b8e-4e4c-4d69-9ff3-b31bbacf9f4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4035060935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":4035060935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":4035060935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4035060935}]},"public_addr":"172.21.15.39:6804/4035060935","cluster_addr":"172.21.15.39:6805/4035060935","heartbeat_back_addr":"172.21.15.39:6807/4035060935","heartbeat_front_addr":"172.21.15.39:6806/4035060935","state":["exists","up"]},{"osd":3,"uuid":"6b725dee-2e37-4fd1-a123-4d38d0e6979e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":4253545661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":4253545661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":4253545661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":4253545661}]},"public_addr":"172.21.15.87:6809/4253545661","cluster_addr":"172.21.15.87:6810/4253545661","heartbeat_back_addr":"172.21.15.87:6812/4253545661","heartbeat_front_addr":"172.21.15.87:6811/4253545661","state":["exists","up"]},{"osd":4,"uuid":"1b99fff1-9eb6-4883-bc7d-7422a005750d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":1078609236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6813","nonce":1078609236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6815","nonce":1078609236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":1078609236}]},"public_addr":"172.21.15.5:6812/1078609236","cluster_addr":"172.21.15.5:6813/1078609236","heartbeat_back_addr":"172.21.15.5:6815/1078609236","heartbeat_front_addr":"172.21.15.5:6814/1078609236","state":["exists","up"]},{"osd":5,"uuid":"148d447f-4841-4fe1-8051-5f3f0094e5d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":702787639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6813","nonce":702787639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6815","nonce":702787639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":702787639}]},"public_addr":"172.21.15.37:6812/702787639","cluster_addr":"172.21.15.37:6813/702787639","heartbeat_back_addr":"172.21.15.37:6815/702787639","heartbeat_front_addr":"172.21.15.37:6814/702787639","state":["exists","up"]},{"osd":6,"uuid":"ac4ec297-ccbb-4bbe-94b3-4859b1a7851a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3381071726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":3381071726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":3381071726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":3381071726}]},"public_addr":"172.21.15.39:6808/3381071726","cluster_addr":"172.21.15.39:6809/3381071726","heartbeat_back_addr":"172.21.15.39:6811/3381071726","heartbeat_front_addr":"172.21.15.39:6810/3381071726","state":["exists","up"]},{"osd":7,"uuid":"f8b5938e-5b74-4ef8-bb86-508b56a05296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":607675738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":607675738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":607675738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":607675738}]},"public_addr":"172.21.15.87:6805/607675738","cluster_addr":"172.21.15.87:6806/607675738","heartbeat_back_addr":"172.21.15.87:6808/607675738","heartbeat_front_addr":"172.21.15.87:6807/607675738","state":["exists","up"]},{"osd":8,"uuid":"407a9a27-7549-489e-8a12-b0951ac5dc26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":608390337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6809","nonce":608390337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6811","nonce":608390337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":608390337}]},"public_addr":"172.21.15.5:6808/608390337","cluster_addr":"172.21.15.5:6809/608390337","heartbeat_back_addr":"172.21.15.5:6811/608390337","heartbeat_front_addr":"172.21.15.5:6810/608390337","state":["exists","up"]},{"osd":9,"uuid":"4f199848-02c4-43ac-8344-36cd6dac6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":3323352543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":3323352543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":3323352543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3323352543}]},"public_addr":"172.21.15.37:6804/3323352543","cluster_addr":"172.21.15.37:6805/3323352543","heartbeat_back_addr":"172.21.15.37:6807/3323352543","heartbeat_front_addr":"172.21.15.37:6806/3323352543","state":["exists","up"]},{"osd":10,"uuid":"dde8ecfc-05bf-47b8-badb-c4f47d8a7b6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":157727954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6801","nonce":157727954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":157727954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":157727954}]},"public_addr":"172.21.15.39:6800/157727954","cluster_addr":"172.21.15.39:6801/157727954","heartbeat_back_addr":"172.21.15.39:6803/157727954","heartbeat_front_addr":"172.21.15.39:6802/157727954","state":["exists","up"]},{"osd":11,"uuid":"52c5aaa9-47d2-41da-a8f2-bb1c69f90259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":1031207741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1031207741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":1031207741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":1031207741}]},"public_addr":"172.21.15.87:6801/1031207741","cluster_addr":"172.21.15.87:6802/1031207741","heartbeat_back_addr":"172.21.15.87:6804/1031207741","heartbeat_front_addr":"172.21.15.87:6803/1031207741","state":["exists","up"]},{"osd":12,"uuid":"96321c3f-1589-4749-ba88-b48cdd72140d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2604688698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6801","nonce":2604688698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6803","nonce":2604688698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2604688698}]},"public_addr":"172.21.15.5:6800/2604688698","cluster_addr":"172.21.15.5:6801/2604688698","heartbeat_back_addr":"172.21.15.5:6803/2604688698","heartbeat_front_addr":"172.21.15.5:6802/2604688698","state":["exists","up"]},{"osd":13,"uuid":"4942dad7-9e60-41a8-8098-12de4684b4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":3207086675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":3207086675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":3207086675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":3207086675}]},"public_addr":"172.21.15.37:6800/3207086675","cluster_addr":"172.21.15.37:6801/3207086675","heartbeat_back_addr":"172.21.15.37:6803/3207086675","heartbeat_front_addr":"172.21.15.37:6802/3207086675","state":["exists","up"]},{"osd":14,"uuid":"1fa4c21c-7e24-49bb-9065-f8a189b9676b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":3204532746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":3204532746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6815","nonce":3204532746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":3204532746}]},"public_addr":"172.21.15.39:6812/3204532746","cluster_addr":"172.21.15.39:6813/3204532746","heartbeat_back_addr":"172.21.15.39:6815/3204532746","heartbeat_front_addr":"172.21.15.39:6814/3204532746","state":["exists","up"]},{"osd":15,"uuid":"aba98af9-0ce7-4292-97fc-f15df5e45ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":2774283835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":2774283835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2774283835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":2774283835}]},"public_addr":"172.21.15.87:6813/2774283835","cluster_addr":"172.21.15.87:6814/2774283835","heartbeat_back_addr":"172.21.15.87:6816/2774283835","heartbeat_front_addr":"172.21.15.87:6815/2774283835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.260097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.771695+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.093611+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.443794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.470232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.458855+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.825491+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.701278+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.240343+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.296272+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.764090+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.749632+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.081066+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:55.712912+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.513217+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T12:01:56.754774+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T12:03:01.937 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T12:03:01.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:02.081 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:02.081 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:02.081 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:02.092 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T12:03:02.390 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T12:03:02.403 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T12:03:02.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:02.531 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:02.531 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:02.531 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:02.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T12:03:02.841 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T12:03:02.854 INFO:tasks.ceph:Scrubbing osd.8 2024-09-13T12:03:02.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:02.916 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:03:02.915+0000 7f4db32f9640 -1 osd.0 73 handle_osd_map injecting map crc failure 2024-09-13T12:03:02.983 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:02.984 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:02.984 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:02.994 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-13T12:03:03.295 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 8 to deep-scrub 2024-09-13T12:03:03.308 INFO:tasks.ceph:Scrubbing osd.12 2024-09-13T12:03:03.309 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:03.437 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:03.438 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:03.438 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:03.449 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-13T12:03:03.625 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:03:03.623+0000 7f4dcc85d640 -1 osd.15 73 handle_osd_map injecting map crc failure 2024-09-13T12:03:03.715 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-09-13T12:03:03.714+0000 7f8c90162640 -1 osd.5 73 handle_osd_map injecting map crc failure 2024-09-13T12:03:03.749 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 12 to deep-scrub 2024-09-13T12:03:03.762 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T12:03:03.762 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:03.799 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:03:03.798+0000 7f1acac3f640 -1 osd.1 73 handle_osd_map injecting map crc failure 2024-09-13T12:03:03.806 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:03:03.805+0000 7fc765526640 -1 osd.4 74 handle_osd_map injecting map crc failure 2024-09-13T12:03:03.892 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:03.893 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:03.893 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:03.903 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T12:03:04.207 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T12:03:04.219 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T12:03:04.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:04.349 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:04.349 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:04.349 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:04.360 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T12:03:04.658 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T12:03:04.672 INFO:tasks.ceph:Scrubbing osd.9 2024-09-13T12:03:04.672 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:04.801 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:04.801 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:04.801 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:04.811 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-13T12:03:05.113 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 9 to deep-scrub 2024-09-13T12:03:05.125 INFO:tasks.ceph:Scrubbing osd.13 2024-09-13T12:03:05.126 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:05.254 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:05.255 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:05.255 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:05.266 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-13T12:03:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 13 to deep-scrub 2024-09-13T12:03:05.583 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T12:03:05.584 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:05.713 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:05.713 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:05.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T12:03:06.024 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T12:03:06.039 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T12:03:06.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:06.169 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:06.169 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:06.169 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:06.179 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T12:03:06.475 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T12:03:06.488 INFO:tasks.ceph:Scrubbing osd.10 2024-09-13T12:03:06.489 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:06.617 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:06.618 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:06.618 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:06.628 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-13T12:03:06.928 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 10 to deep-scrub 2024-09-13T12:03:06.941 INFO:tasks.ceph:Scrubbing osd.14 2024-09-13T12:03:06.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:07.070 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:07.071 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:07.071 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:07.081 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-13T12:03:07.306 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:03:07.305+0000 7fc765526640 -1 osd.4 75 handle_osd_map injecting map crc failure 2024-09-13T12:03:07.382 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 14 to deep-scrub 2024-09-13T12:03:07.395 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T12:03:07.395 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:07.524 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:07.525 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:07.525 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:07.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T12:03:07.696 INFO:tasks.ceph.osd.9.smithi037.stderr:2024-09-13T12:03:07.695+0000 7ff4e5f93640 -1 osd.9 75 handle_osd_map injecting map crc failure 2024-09-13T12:03:07.806 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-09-13T12:03:07.805+0000 7f1acac3f640 -1 osd.1 75 handle_osd_map injecting map crc failure 2024-09-13T12:03:07.806 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:03:07.805+0000 7f328fcee640 -1 osd.13 75 handle_osd_map injecting map crc failure 2024-09-13T12:03:07.833 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T12:03:07.848 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T12:03:07.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:07.977 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:07.977 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:07.977 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:07.985 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:03:07.985+0000 7fbe39fa0640 -1 osd.2 75 handle_osd_map injecting map crc failure 2024-09-13T12:03:07.987 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T12:03:08.290 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T12:03:08.303 INFO:tasks.ceph:Scrubbing osd.11 2024-09-13T12:03:08.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:08.431 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:08.432 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:08.432 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:08.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-13T12:03:08.746 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 11 to deep-scrub 2024-09-13T12:03:08.759 INFO:tasks.ceph:Scrubbing osd.15 2024-09-13T12:03:08.759 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-13T12:03:08.887 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-09-13T12:03:08.887 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T12:03:08.887 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-09-13T12:03:08.897 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-13T12:03:09.197 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 15 to deep-scrub 2024-09-13T12:03:09.209 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:03:09.509 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:03:09.510 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:03:09.522 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-13T12:03:09.067299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":455928,"kb_used_data":10972,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493512,"statfs":{"total":1546188226560,"available":1545721356288,"internally_reserved":0,"allocated":11235328,"data_stored":7700340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.050159"},"pg_stats":[{"pgid":"1.0","version":"25'112","reported_seq":244,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.801671+0000","last_change":"2024-09-13T12:01:59.547631+0000","last_active":"2024-09-13T12:03:03.801671+0000","last_peered":"2024-09-13T12:03:03.801671+0000","last_clean":"2024-09-13T12:03:03.801671+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:03:03.801671+0000","last_undegraded":"2024-09-13T12:03:03.801671+0000","last_fullsized":"2024-09-13T12:03:03.801671+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:01:58.267157+0000","last_clean_scrub_stamp":"2024-09-13T12:01:58.267157+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:27.903189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":108,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:04.085709+0000","last_change":"2024-09-13T12:02:03.357455+0000","last_active":"2024-09-13T12:03:04.085709+0000","last_peered":"2024-09-13T12:03:04.085709+0000","last_clean":"2024-09-13T12:03:04.085709+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:03:04.085709+0000","last_undegraded":"2024-09-13T12:03:04.085709+0000","last_fullsized":"2024-09-13T12:03:04.085709+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:08.173291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":96,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:04.085687+0000","last_change":"2024-09-13T12:02:13.386777+0000","last_active":"2024-09-13T12:03:04.085687+0000","last_peered":"2024-09-13T12:03:04.085687+0000","last_clean":"2024-09-13T12:03:04.085687+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:03:04.085687+0000","last_undegraded":"2024-09-13T12:03:04.085687+0000","last_fullsized":"2024-09-13T12:03:04.085687+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.853734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":108,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.803139+0000","last_change":"2024-09-13T12:02:04.493297+0000","last_active":"2024-09-13T12:03:03.803139+0000","last_peered":"2024-09-13T12:03:03.803139+0000","last_clean":"2024-09-13T12:03:03.803139+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:03:03.803139+0000","last_undegraded":"2024-09-13T12:03:03.803139+0000","last_fullsized":"2024-09-13T12:03:03.803139+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:24.721087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":110,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T12:03:07.308190+0000","last_change":"2024-09-13T12:03:04.933874+0000","last_active":"2024-09-13T12:03:07.308190+0000","last_peered":"2024-09-13T12:03:07.308190+0000","last_clean":"2024-09-13T12:03:07.308190+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:03:07.308190+0000","last_undegraded":"2024-09-13T12:03:07.308190+0000","last_fullsized":"2024-09-13T12:03:07.308190+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:04.933797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:04.933797+0000","last_clean_scrub_stamp":"2024-09-13T12:03:04.933797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:09.542909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":115,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:04.085683+0000","last_change":"2024-09-13T12:03:02.673110+0000","last_active":"2024-09-13T12:03:04.085683+0000","last_peered":"2024-09-13T12:03:04.085683+0000","last_clean":"2024-09-13T12:03:04.085683+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:03:04.085683+0000","last_undegraded":"2024-09-13T12:03:04.085683+0000","last_fullsized":"2024-09-13T12:03:04.085683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:02.673018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:03:02.673018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:18.657304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":98,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.202558+0000","last_change":"2024-09-13T12:02:13.558315+0000","last_active":"2024-09-13T12:03:03.202558+0000","last_peered":"2024-09-13T12:03:03.202558+0000","last_clean":"2024-09-13T12:03:03.202558+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:03:03.202558+0000","last_undegraded":"2024-09-13T12:03:03.202558+0000","last_fullsized":"2024-09-13T12:03:03.202558+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:12.366121+0000","last_clean_scrub_stamp":"2024-09-13T12:02:12.366121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:25.911951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":109,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.023418+0000","last_change":"2024-09-13T12:02:03.326392+0000","last_active":"2024-09-13T12:03:03.023418+0000","last_peered":"2024-09-13T12:03:03.023418+0000","last_clean":"2024-09-13T12:03:03.023418+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:03:03.023418+0000","last_undegraded":"2024-09-13T12:03:03.023418+0000","last_fullsized":"2024-09-13T12:03:03.023418+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.987604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":109,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T12:03:07.908546+0000","last_change":"2024-09-13T12:03:07.908546+0000","last_active":"2024-09-13T12:03:07.908546+0000","last_peered":"2024-09-13T12:03:07.908546+0000","last_clean":"2024-09-13T12:03:07.908546+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:03:07.908546+0000","last_undegraded":"2024-09-13T12:03:07.908546+0000","last_fullsized":"2024-09-13T12:03:07.908546+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:07.908452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:07.908452+0000","last_clean_scrub_stamp":"2024-09-13T12:03:07.908452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:20.798301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":97,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.801724+0000","last_change":"2024-09-13T12:02:15.395418+0000","last_active":"2024-09-13T12:03:03.801724+0000","last_peered":"2024-09-13T12:03:03.801724+0000","last_clean":"2024-09-13T12:03:03.801724+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:03:03.801724+0000","last_undegraded":"2024-09-13T12:03:03.801724+0000","last_fullsized":"2024-09-13T12:03:03.801724+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.390597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":121,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T12:03:07.308197+0000","last_change":"2024-09-13T12:03:02.917143+0000","last_active":"2024-09-13T12:03:07.308197+0000","last_peered":"2024-09-13T12:03:07.308197+0000","last_clean":"2024-09-13T12:03:07.308197+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:03:07.308197+0000","last_undegraded":"2024-09-13T12:03:07.308197+0000","last_fullsized":"2024-09-13T12:03:07.308197+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:02.917073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:02.917073+0000","last_clean_scrub_stamp":"2024-09-13T12:03:02.917073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:03.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":96,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.023708+0000","last_change":"2024-09-13T12:02:15.533754+0000","last_active":"2024-09-13T12:03:03.023708+0000","last_peered":"2024-09-13T12:03:03.023708+0000","last_clean":"2024-09-13T12:03:03.023708+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:03:03.023708+0000","last_undegraded":"2024-09-13T12:03:03.023708+0000","last_fullsized":"2024-09-13T12:03:03.023708+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:32.750884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":106,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.801587+0000","last_change":"2024-09-13T12:02:03.358118+0000","last_active":"2024-09-13T12:03:03.801587+0000","last_peered":"2024-09-13T12:03:03.801587+0000","last_clean":"2024-09-13T12:03:03.801587+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:03:03.801587+0000","last_undegraded":"2024-09-13T12:03:03.801587+0000","last_fullsized":"2024-09-13T12:03:03.801587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:28.792957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":111,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T12:03:07.308190+0000","last_change":"2024-09-13T12:03:03.901280+0000","last_active":"2024-09-13T12:03:07.308190+0000","last_peered":"2024-09-13T12:03:07.308190+0000","last_clean":"2024-09-13T12:03:07.308190+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:03:07.308190+0000","last_undegraded":"2024-09-13T12:03:07.308190+0000","last_fullsized":"2024-09-13T12:03:07.308190+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:03.901217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:03.901217+0000","last_clean_scrub_stamp":"2024-09-13T12:03:03.901217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:31.836536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":119,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T12:03:07.039005+0000","last_change":"2024-09-13T12:03:03.997634+0000","last_active":"2024-09-13T12:03:07.039005+0000","last_peered":"2024-09-13T12:03:07.039005+0000","last_clean":"2024-09-13T12:03:07.039005+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:03:07.039005+0000","last_undegraded":"2024-09-13T12:03:07.039005+0000","last_fullsized":"2024-09-13T12:03:07.039005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:03.997561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:03.997561+0000","last_clean_scrub_stamp":"2024-09-13T12:03:03.997561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:28.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":96,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.023701+0000","last_change":"2024-09-13T12:02:15.395347+0000","last_active":"2024-09-13T12:03:03.023701+0000","last_peered":"2024-09-13T12:03:03.023701+0000","last_clean":"2024-09-13T12:03:03.023701+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:03:03.023701+0000","last_undegraded":"2024-09-13T12:03:03.023701+0000","last_fullsized":"2024-09-13T12:03:03.023701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:14.379019+0000","last_clean_scrub_stamp":"2024-09-13T12:02:14.379019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:16.264185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-09-13T12:03:03.023450+0000","last_change":"2024-09-13T12:02:03.325780+0000","last_active":"2024-09-13T12:03:03.023450+0000","last_peered":"2024-09-13T12:03:03.023450+0000","last_clean":"2024-09-13T12:03:03.023450+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:03:03.023450+0000","last_undegraded":"2024-09-13T12:03:03.023450+0000","last_fullsized":"2024-09-13T12:03:03.023450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:02:00.295473+0000","last_clean_scrub_stamp":"2024-09-13T12:02:00.295473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:03:14.707022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247826,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":413441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247824,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1138688,"data_stored":927403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31788,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":618496,"data_stored":394646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":643072,"data_stored":416084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":420029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":413440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280530,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":651264,"data_stored":423480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31828,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340012,"statfs":{"total":96636764160,"available":96604172288,"internally_reserved":0,"allocated":659456,"data_stored":424672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247827,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":398097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247827,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":413083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":630784,"data_stored":413441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":411153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247827,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1228800,"data_stored":999192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247826,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":400323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280529,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":663552,"data_stored":424592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247825,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":407264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:03:09.526 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T12:01:58.267157+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=1, tm_sec=58, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.526 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T12:02:00.295473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.527 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T12:02:12.366121+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=12, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.527 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T12:02:00.295473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.527 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-13T12:02:12.366121+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=12, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.527 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T12:02:00.295473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.527 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-13T12:02:14.379019+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=14, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.528 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-13T12:02:14.379019+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=14, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.528 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T12:02:00.295473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.528 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-13T12:02:14.379019+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=14, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.528 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T12:02:00.295473+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=2, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=12, tm_min=3, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T12:03:09.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T12:03:10.053 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:03:10.052+0000 7f4dca859640 -1 osd.15 76 handle_osd_map injecting map crc failure 2024-09-13T12:03:15.094 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:03:15.093+0000 7fbe3bfa4640 -1 osd.2 77 handle_osd_map injecting map crc failure 2024-09-13T12:03:17.206 INFO:tasks.ceph.osd.8.smithi005.stderr:2024-09-13T12:03:17.206+0000 7f5b85735640 -1 osd.8 78 handle_osd_map injecting map crc failure 2024-09-13T12:03:17.308 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:03:17.307+0000 7f374f3c5640 -1 osd.12 78 handle_osd_map injecting map crc failure 2024-09-13T12:03:17.478 INFO:tasks.ceph.osd.14.smithi039.stderr:2024-09-13T12:03:17.477+0000 7f63ff314640 -1 osd.14 77 handle_osd_map injecting map crc failure 2024-09-13T12:03:18.901 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:03:18.900+0000 7f932ce94640 -1 osd.6 79 handle_osd_map injecting map crc failure 2024-09-13T12:03:19.768 INFO:tasks.ceph.osd.7.smithi087.stderr:2024-09-13T12:03:19.767+0000 7fa055760640 -1 osd.7 80 handle_osd_map injecting map crc failure 2024-09-13T12:03:22.668 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-09-13T12:03:22.667+0000 7f4db32f9640 -1 osd.0 81 handle_osd_map injecting map crc failure 2024-09-13T12:03:24.134 INFO:tasks.ceph.osd.13.smithi037.stderr:2024-09-13T12:03:24.133+0000 7f3291cf2640 -1 osd.13 82 handle_osd_map injecting map crc failure 2024-09-13T12:03:24.702 INFO:tasks.ceph.osd.15.smithi087.stderr:2024-09-13T12:03:24.701+0000 7f4dca859640 -1 osd.15 82 handle_osd_map injecting map crc failure 2024-09-13T12:03:24.702 INFO:tasks.ceph.osd.11.smithi087.stderr:2024-09-13T12:03:24.701+0000 7f36821c9640 -1 osd.11 82 handle_osd_map injecting map crc failure 2024-09-13T12:03:26.410 INFO:tasks.ceph.osd.4.smithi005.stderr:2024-09-13T12:03:26.409+0000 7fc765526640 -1 osd.4 83 handle_osd_map injecting map crc failure 2024-09-13T12:03:27.369 INFO:tasks.ceph.osd.12.smithi005.stderr:2024-09-13T12:03:27.367+0000 7f374f3c5640 -1 osd.12 84 handle_osd_map injecting map crc failure 2024-09-13T12:03:27.614 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-13T12:03:27.613+0000 7fbe39fa0640 -1 osd.2 84 handle_osd_map injecting map crc failure 2024-09-13T12:03:27.615 INFO:tasks.ceph.osd.6.smithi039.stderr:2024-09-13T12:03:27.613+0000 7f932ce94640 -1 osd.6 84 handle_osd_map injecting map crc failure 2024-09-13T12:03:29.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T12:03:29.846 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-09-13T12:03:29.846 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-09-13T12:03:29.858 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-13T12:03:29.581284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":456188,"kb_used_data":10940,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509493252,"statfs":{"total":1546188226560,"available":1545721090048,"internally_reserved":0,"allocated":11202560,"data_stored":7735343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25520,"internal_metadata":455253072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001280"},"pg_stats":[{"pgid":"1.0","version":"76'114","reported_seq":274,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.618607+0000","last_change":"2024-09-13T12:03:07.197552+0000","last_active":"2024-09-13T12:03:27.618607+0000","last_peered":"2024-09-13T12:03:27.618607+0000","last_clean":"2024-09-13T12:03:27.618607+0000","last_became_active":"2024-09-13T12:01:59.546709+0000","last_became_peered":"2024-09-13T12:01:59.546709+0000","last_unstale":"2024-09-13T12:03:27.618607+0000","last_undegraded":"2024-09-13T12:03:27.618607+0000","last_fullsized":"2024-09-13T12:03:27.618607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'114","last_scrub_stamp":"2024-09-13T12:03:07.197468+0000","last_deep_scrub":"76'114","last_deep_scrub_stamp":"2024-09-13T12:03:07.197468+0000","last_clean_scrub_stamp":"2024-09-13T12:03:07.197468+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:35.323092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,15],"acting":[6,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"77'2","reported_seq":135,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.147354+0000","last_change":"2024-09-13T12:03:09.659144+0000","last_active":"2024-09-13T12:03:27.147354+0000","last_peered":"2024-09-13T12:03:27.147354+0000","last_clean":"2024-09-13T12:03:27.147354+0000","last_became_active":"2024-09-13T12:02:01.601348+0000","last_became_peered":"2024-09-13T12:02:01.601348+0000","last_unstale":"2024-09-13T12:03:27.147354+0000","last_undegraded":"2024-09-13T12:03:27.147354+0000","last_fullsized":"2024-09-13T12:03:27.147354+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'2","last_scrub_stamp":"2024-09-13T12:03:09.659033+0000","last_deep_scrub":"77'2","last_deep_scrub_stamp":"2024-09-13T12:03:09.659033+0000","last_clean_scrub_stamp":"2024-09-13T12:03:09.659033+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:36.044983+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00039995699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,5],"acting":[14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":122,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.147338+0000","last_change":"2024-09-13T12:03:10.644391+0000","last_active":"2024-09-13T12:03:27.147338+0000","last_peered":"2024-09-13T12:03:27.147338+0000","last_clean":"2024-09-13T12:03:27.147338+0000","last_became_active":"2024-09-13T12:02:13.386448+0000","last_became_peered":"2024-09-13T12:02:13.386448+0000","last_unstale":"2024-09-13T12:03:27.147338+0000","last_undegraded":"2024-09-13T12:03:27.147338+0000","last_fullsized":"2024-09-13T12:03:27.147338+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:10.644282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:10.644282+0000","last_clean_scrub_stamp":"2024-09-13T12:03:10.644282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:32.758236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3],"acting":[14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.404154+0000","last_change":"2024-09-13T12:03:06.047699+0000","last_active":"2024-09-13T12:03:27.404154+0000","last_peered":"2024-09-13T12:03:27.404154+0000","last_clean":"2024-09-13T12:03:27.404154+0000","last_became_active":"2024-09-13T12:02:01.439732+0000","last_became_peered":"2024-09-13T12:02:01.439732+0000","last_unstale":"2024-09-13T12:03:27.404154+0000","last_undegraded":"2024-09-13T12:03:27.404154+0000","last_fullsized":"2024-09-13T12:03:27.404154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:06.047616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:06.047616+0000","last_clean_scrub_stamp":"2024-09-13T12:03:06.047616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:15.630018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000527357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3],"acting":[9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.610472+0000","last_change":"2024-09-13T12:03:04.933874+0000","last_active":"2024-09-13T12:03:27.610472+0000","last_peered":"2024-09-13T12:03:27.610472+0000","last_clean":"2024-09-13T12:03:27.610472+0000","last_became_active":"2024-09-13T12:02:13.385436+0000","last_became_peered":"2024-09-13T12:02:13.385436+0000","last_unstale":"2024-09-13T12:03:27.610472+0000","last_undegraded":"2024-09-13T12:03:27.610472+0000","last_fullsized":"2024-09-13T12:03:27.610472+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:04.933797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:04.933797+0000","last_clean_scrub_stamp":"2024-09-13T12:03:04.933797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:09.542909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,13],"acting":[4,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":141,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.147329+0000","last_change":"2024-09-13T12:03:08.646094+0000","last_active":"2024-09-13T12:03:27.147329+0000","last_peered":"2024-09-13T12:03:27.147329+0000","last_clean":"2024-09-13T12:03:27.147329+0000","last_became_active":"2024-09-13T12:02:01.359894+0000","last_became_peered":"2024-09-13T12:02:01.359894+0000","last_unstale":"2024-09-13T12:03:27.147329+0000","last_undegraded":"2024-09-13T12:03:27.147329+0000","last_fullsized":"2024-09-13T12:03:27.147329+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:08.645997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:08.645997+0000","last_clean_scrub_stamp":"2024-09-13T12:03:08.645997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:14.256586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0],"acting":[14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":126,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.145657+0000","last_change":"2024-09-13T12:03:08.523975+0000","last_active":"2024-09-13T12:03:27.145657+0000","last_peered":"2024-09-13T12:03:27.145657+0000","last_clean":"2024-09-13T12:03:27.145657+0000","last_became_active":"2024-09-13T12:02:13.558065+0000","last_became_peered":"2024-09-13T12:02:13.558065+0000","last_unstale":"2024-09-13T12:03:27.145657+0000","last_undegraded":"2024-09-13T12:03:27.145657+0000","last_fullsized":"2024-09-13T12:03:27.145657+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:08.523880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:08.523880+0000","last_clean_scrub_stamp":"2024-09-13T12:03:08.523880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:15.589323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"75'3","reported_seq":138,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.613568+0000","last_change":"2024-09-13T12:03:05.697728+0000","last_active":"2024-09-13T12:03:27.613568+0000","last_peered":"2024-09-13T12:03:27.613568+0000","last_clean":"2024-09-13T12:03:27.613568+0000","last_became_active":"2024-09-13T12:02:01.656447+0000","last_became_peered":"2024-09-13T12:02:01.656447+0000","last_unstale":"2024-09-13T12:03:27.613568+0000","last_undegraded":"2024-09-13T12:03:27.613568+0000","last_fullsized":"2024-09-13T12:03:27.613568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'3","last_scrub_stamp":"2024-09-13T12:03:05.697641+0000","last_deep_scrub":"75'3","last_deep_scrub_stamp":"2024-09-13T12:03:05.697641+0000","last_clean_scrub_stamp":"2024-09-13T12:03:05.697641+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:11.238520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001089561,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12],"acting":[13,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.186476+0000","last_change":"2024-09-13T12:03:07.908546+0000","last_active":"2024-09-13T12:03:27.186476+0000","last_peered":"2024-09-13T12:03:27.186476+0000","last_clean":"2024-09-13T12:03:27.186476+0000","last_became_active":"2024-09-13T12:02:13.469818+0000","last_became_peered":"2024-09-13T12:02:13.469818+0000","last_unstale":"2024-09-13T12:03:27.186476+0000","last_undegraded":"2024-09-13T12:03:27.186476+0000","last_fullsized":"2024-09-13T12:03:27.186476+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:07.908452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:07.908452+0000","last_clean_scrub_stamp":"2024-09-13T12:03:07.908452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:20.798301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9],"acting":[10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":125,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.618613+0000","last_change":"2024-09-13T12:03:08.197426+0000","last_active":"2024-09-13T12:03:27.618613+0000","last_peered":"2024-09-13T12:03:27.618613+0000","last_clean":"2024-09-13T12:03:27.618613+0000","last_became_active":"2024-09-13T12:02:15.395116+0000","last_became_peered":"2024-09-13T12:02:15.395116+0000","last_unstale":"2024-09-13T12:03:27.618613+0000","last_undegraded":"2024-09-13T12:03:27.618613+0000","last_fullsized":"2024-09-13T12:03:27.618613+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:08.197338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:08.197338+0000","last_clean_scrub_stamp":"2024-09-13T12:03:08.197338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:27.893075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.610474+0000","last_change":"2024-09-13T12:03:02.917143+0000","last_active":"2024-09-13T12:03:27.610474+0000","last_peered":"2024-09-13T12:03:27.610474+0000","last_clean":"2024-09-13T12:03:27.610474+0000","last_became_active":"2024-09-13T12:02:01.359656+0000","last_became_peered":"2024-09-13T12:02:01.359656+0000","last_unstale":"2024-09-13T12:03:27.610474+0000","last_undegraded":"2024-09-13T12:03:27.610474+0000","last_fullsized":"2024-09-13T12:03:27.610474+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:02.917073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:02.917073+0000","last_clean_scrub_stamp":"2024-09-13T12:03:02.917073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:03.063204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044011799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":124,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.618565+0000","last_change":"2024-09-13T12:03:06.992686+0000","last_active":"2024-09-13T12:03:27.618565+0000","last_peered":"2024-09-13T12:03:27.618565+0000","last_clean":"2024-09-13T12:03:27.618565+0000","last_became_active":"2024-09-13T12:02:15.533485+0000","last_became_peered":"2024-09-13T12:02:15.533485+0000","last_unstale":"2024-09-13T12:03:27.618565+0000","last_undegraded":"2024-09-13T12:03:27.618565+0000","last_fullsized":"2024-09-13T12:03:27.618565+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:06.992603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:06.992603+0000","last_clean_scrub_stamp":"2024-09-13T12:03:06.992603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:23.966739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":134,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.274352+0000","last_change":"2024-09-13T12:03:08.581149+0000","last_active":"2024-09-13T12:03:27.274352+0000","last_peered":"2024-09-13T12:03:27.274352+0000","last_clean":"2024-09-13T12:03:27.274352+0000","last_became_active":"2024-09-13T12:02:01.313977+0000","last_became_peered":"2024-09-13T12:02:01.313977+0000","last_unstale":"2024-09-13T12:03:27.274352+0000","last_undegraded":"2024-09-13T12:03:27.274352+0000","last_fullsized":"2024-09-13T12:03:27.274352+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:08.581060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:08.581060+0000","last_clean_scrub_stamp":"2024-09-13T12:03:08.581060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:09.295431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00038758899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.610441+0000","last_change":"2024-09-13T12:03:03.901280+0000","last_active":"2024-09-13T12:03:27.610441+0000","last_peered":"2024-09-13T12:03:27.610441+0000","last_clean":"2024-09-13T12:03:27.610441+0000","last_became_active":"2024-09-13T12:02:15.393748+0000","last_became_peered":"2024-09-13T12:02:15.393748+0000","last_unstale":"2024-09-13T12:03:27.610441+0000","last_undegraded":"2024-09-13T12:03:27.610441+0000","last_fullsized":"2024-09-13T12:03:27.610441+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:03.901217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:03.901217+0000","last_clean_scrub_stamp":"2024-09-13T12:03:03.901217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:31.836536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9],"acting":[4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.145089+0000","last_change":"2024-09-13T12:03:03.997634+0000","last_active":"2024-09-13T12:03:27.145089+0000","last_peered":"2024-09-13T12:03:27.145089+0000","last_clean":"2024-09-13T12:03:27.145089+0000","last_became_active":"2024-09-13T12:02:01.660098+0000","last_became_peered":"2024-09-13T12:02:01.660098+0000","last_unstale":"2024-09-13T12:03:27.145089+0000","last_undegraded":"2024-09-13T12:03:27.145089+0000","last_fullsized":"2024-09-13T12:03:27.145089+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:03.997561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:03.997561+0000","last_clean_scrub_stamp":"2024-09-13T12:03:03.997561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:28.299514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051923099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11],"acting":[8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":124,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.618538+0000","last_change":"2024-09-13T12:03:07.986300+0000","last_active":"2024-09-13T12:03:27.618538+0000","last_peered":"2024-09-13T12:03:27.618538+0000","last_clean":"2024-09-13T12:03:27.618538+0000","last_became_active":"2024-09-13T12:02:15.395051+0000","last_became_peered":"2024-09-13T12:02:15.395051+0000","last_unstale":"2024-09-13T12:03:27.618538+0000","last_undegraded":"2024-09-13T12:03:27.618538+0000","last_fullsized":"2024-09-13T12:03:27.618538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:07.986212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:07.986212+0000","last_clean_scrub_stamp":"2024-09-13T12:03:07.986212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:23.998742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T12:03:27.613568+0000","last_change":"2024-09-13T12:03:06.690792+0000","last_active":"2024-09-13T12:03:27.613568+0000","last_peered":"2024-09-13T12:03:27.613568+0000","last_clean":"2024-09-13T12:03:27.613568+0000","last_became_active":"2024-09-13T12:02:01.664682+0000","last_became_peered":"2024-09-13T12:02:01.664682+0000","last_unstale":"2024-09-13T12:03:27.613568+0000","last_undegraded":"2024-09-13T12:03:27.613568+0000","last_fullsized":"2024-09-13T12:03:27.613568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T12:03:06.690690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T12:03:06.690690+0000","last_clean_scrub_stamp":"2024-09-13T12:03:06.690690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T12:04:12.491836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041616199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10],"acting":[13,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":5},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":1119296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":9,"up_from":23,"seq":98784247831,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":410439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":23,"seq":98784247829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1142784,"data_stored":937741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":23,"seq":98784247830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":634880,"data_stored":410767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313238,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":733184,"data_stored":477556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":409357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,7,8,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":409778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":402790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":410291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247832,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":409997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247832,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":409149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":622592,"data_stored":410145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":23,"seq":98784247830,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":407857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247832,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":1000273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247831,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":409787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,6,8,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":410161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247830,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":610304,"data_stored":409255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T12:03:29.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T12:03:30.176 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T12:03:30.180 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T12:03:30.181 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T12:03:30.181 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.224 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T12:03:30.224 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T12:03:30.224 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.268 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T12:03:30.269 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-13T12:03:30.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.299 INFO:tasks.ceph.osd.8:Stopped 2024-09-13T12:03:30.299 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-13T12:03:30.299 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.369 INFO:tasks.ceph.osd.12:Stopped 2024-09-13T12:03:30.369 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T12:03:30.369 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.402 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T12:03:30.402 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T12:03:30.403 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.482 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T12:03:30.482 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-13T12:03:30.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.551 INFO:tasks.ceph.osd.9:Stopped 2024-09-13T12:03:30.552 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-13T12:03:30.552 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.622 INFO:tasks.ceph.osd.13:Stopped 2024-09-13T12:03:30.622 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T12:03:30.622 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.654 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T12:03:30.655 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T12:03:30.655 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.728 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T12:03:30.729 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-13T12:03:30.729 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.804 INFO:tasks.ceph.osd.10:Stopped 2024-09-13T12:03:30.805 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-13T12:03:30.805 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.878 INFO:tasks.ceph.osd.14:Stopped 2024-09-13T12:03:30.879 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T12:03:30.879 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.913 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T12:03:30.914 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T12:03:30.914 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:30.981 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T12:03:30.982 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-13T12:03:30.982 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.047 INFO:tasks.ceph.osd.11:Stopped 2024-09-13T12:03:31.047 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-13T12:03:31.047 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.121 INFO:tasks.ceph.osd.15:Stopped 2024-09-13T12:03:31.121 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T12:03:31.122 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T12:03:31.122 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.147 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T12:03:31.148 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T12:03:31.148 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.194 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T12:03:31.194 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T12:03:31.194 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T12:03:31.195 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.206 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T12:03:31.207 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T12:03:31.207 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.221 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T12:03:31.221 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T12:03:31.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T12:03:31.235 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T12:03:31.235 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T12:03:31.236 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'must scrub before tier agent can activate' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T12:03:31.288 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:03:31.289 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T12:03:31.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:03:31.536 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T12:03:31.612 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:03:31.613 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-13T12:03:31.676 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:03:31.676 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-13T12:03:31.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:03:31.741 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T12:03:31.853 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:03:31.854 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T12:03:31.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:03:31.977 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-13T12:03:32.035 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:03:32.036 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-13T12:03:32.092 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:03:32.093 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T12:03:32.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:03:32.215 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T12:03:32.274 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:03:32.275 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-13T12:03:32.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:03:32.329 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-13T12:03:32.379 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:03:32.379 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T12:03:32.674 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:03:32.675 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T12:03:32.752 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:03:32.753 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-13T12:03:32.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:03:32.810 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-13T12:03:32.867 INFO:tasks.ceph:Archiving mon data... 2024-09-13T12:03:32.868 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/data/mon.a.tgz 2024-09-13T12:03:32.868 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-09-13T12:03:32.887 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.HFss2jC7SD 2024-09-13T12:03:32.887 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HFss2jC7SD 2024-09-13T12:03:33.040 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.HFss2jC7SD 2024-09-13T12:03:33.130 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.HFss2jC7SD is 362KB 2024-09-13T12:03:33.229 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.HFss2jC7SD 2024-09-13T12:03:33.245 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/data/mon.b.tgz 2024-09-13T12:03:33.245 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-09-13T12:03:33.265 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.btjNzOh4aS 2024-09-13T12:03:33.266 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.btjNzOh4aS 2024-09-13T12:03:33.411 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.btjNzOh4aS 2024-09-13T12:03:33.497 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.btjNzOh4aS is 369KB 2024-09-13T12:03:33.568 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.btjNzOh4aS 2024-09-13T12:03:33.584 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/data/mon.c.tgz 2024-09-13T12:03:33.584 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-13T12:03:33.603 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.wzNaAZJtTE 2024-09-13T12:03:33.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wzNaAZJtTE 2024-09-13T12:03:33.753 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.wzNaAZJtTE 2024-09-13T12:03:33.840 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.wzNaAZJtTE is 369KB 2024-09-13T12:03:33.910 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.wzNaAZJtTE 2024-09-13T12:03:33.925 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T12:03:33.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T12:03:33.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T12:03:33.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T12:03:33.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T12:03:34.015 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T12:03:34.033 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T12:03:34.049 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T12:03:34.067 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T12:03:34.067 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T12:03:34.072 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi005/crash 2024-09-13T12:03:34.073 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T12:03:34.103 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi037/crash 2024-09-13T12:03:34.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T12:03:34.132 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi039/crash 2024-09-13T12:03:34.133 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T12:03:34.159 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi087/crash 2024-09-13T12:03:34.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T12:03:34.281 INFO:tasks.ceph:Compressing logs... 2024-09-13T12:03:34.282 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T12:03:34.284 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T12:03:34.287 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T12:03:34.290 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T12:03:34.313 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T12:03:34.313 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-09-13T12:03:34.313 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-13T12:03:34.319 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T12:03:34.320 INFO:teuthology.orchestra.run.smithi039.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T12:03:34.320 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.log: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T12:03:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T12:03:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T12:03:34.321 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T12:03:34.342 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32981.log 2024-09-13T12:03:34.342 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38159.log 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log/var/log/ceph/ceph-client.admin.32981.log: 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32981.log.gz 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32701.log 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38159.log.gz 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32781.log 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32941.log 2024-09-13T12:03:34.343 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32701.log.gz 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32781.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32781.log.gz 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32621.log 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32941.log.gz 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32901.log 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32621.log.gz 2024-09-13T12:03:34.344 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33021.log 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32901.log.gz 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32494.log 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.33021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38075.log 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33021.log.gz 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32494.log.gz 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32861.log 2024-09-13T12:03:34.345 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38075.log.gz 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38713.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32861.log.gz 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37991.log 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32581.log 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2024-09-13T12:03:34.346 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.37991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37991.log.gz 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32581.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.32581.log.gz 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-client.admin.38117.log 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38525.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz -- 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/ceph-osd.11.log 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32661.log 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38117.log.gz 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38033.log 2024-09-13T12:03:34.347 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32661.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32661.log.gz 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38033.log.gz 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32410.log 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.38221.log 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32741.log 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32410.log.gz 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.38221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-09-13T12:03:34.348 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32821.log 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32450.log 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32741.log.gz 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32821.log.gz 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32541.log 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32450.log.gz 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-client.admin.32541.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.32541.log.gz --verbose 2024-09-13T12:03:34.349 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/ceph-client.admin.38423.log 2024-09-13T12:03:34.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.38423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-09-13T12:03:34.385 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T12:03:34.407 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44090.log 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42401.log 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44090.log.gz 2024-09-13T12:03:34.408 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42401.log.gz 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39351.log 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38264.log 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42485.log 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39351.log.gz 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38264.log.gz 2024-09-13T12:03:34.409 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41981.log 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42485.log.gz 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41981.log.gz 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45425.log 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-09-13T12:03:34.410 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43449.log 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45425.log.gz 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43449.log.gz 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45719.log 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43453.log 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-09-13T12:03:34.411 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-09-13T12:03:34.412 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45719.log.gz 2024-09-13T12:03:34.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43457.log 2024-09-13T12:03:34.412 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43453.log.gz 2024-09-13T12:03:34.412 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-09-13T12:03:34.412 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38259.log 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43457.log.gz 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38259.log.gz 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2024-09-13T12:03:34.413 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43878.log 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40815.log 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43878.log.gz 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40815.log.gz 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38798.log 2024-09-13T12:03:34.414 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38256.log 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45971.log 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38798.log.gz 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42569.log 2024-09-13T12:03:34.415 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45971.log.gz 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44127.log 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42569.log.gz 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44127.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44127.log.gz 2024-09-13T12:03:34.416 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45383.log 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43446.log 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45383.log.gz 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2024-09-13T12:03:34.417 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43446.log.gz 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45761.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27626.log 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.tmp-client.admin.27626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44837.log 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27626.log.gz 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-09-13T12:03:34.418 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42275.log 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44837.log.gz 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42275.log.gzgzip 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44138.log 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42905.log 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44138.log.gz 2024-09-13T12:03:34.419 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46055.log 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44111.log 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42905.log.gz 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46055.log.gz 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45173.log 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33768.log 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44111.log.gz 2024-09-13T12:03:34.420 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45173.log.gz 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.33768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33768.log.gz 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2024-09-13T12:03:34.421 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38867.log 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38776.log 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38867.log.gz 2024-09-13T12:03:34.422 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41874.log 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43440.log 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41874.log.gz 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43440.log.gz 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45131.log 2024-09-13T12:03:34.423 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40897.log 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42863.log 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45131.log.gz 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37883.log 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr: 17.9% -- replaced with /var/log/ceph/ceph-client.admin.40897.log.gz 2024-09-13T12:03:34.424 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42863.log.gz 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43419.log 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37883.log.gz 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42107.log 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43419.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.43419.log.gz /var/log/ceph/ceph-client.admin.39647.log 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T12:03:34.425 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42107.log.gz 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38263.log 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39311.log 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38263.log.gz 2024-09-13T12:03:34.426 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39311.log.gz 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42191.log 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz -- 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.46097.log 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42191.log.gz 2024-09-13T12:03:34.427 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40735.log 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44110.log 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gz 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40735.log.gz 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38233.log 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46221.log 2024-09-13T12:03:34.428 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44110.log.gz 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38233.log.gz 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46221.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46221.log.gz -5 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44963.log 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-09-13T12:03:34.429 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44963.log.gz 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-09-13T12:03:34.430 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43969.log 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41367.log 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43418.log 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43969.log.gz 2024-09-13T12:03:34.431 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr: 17.2% -- replaced with /var/log/ceph/ceph-client.admin.41367.log.gz 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43437.log 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43418.log.gz 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41686.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz -5 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37627.log 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43437.log.gz 2024-09-13T12:03:34.432 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37627.log.gz 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38649.log 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38649.log.gz 2024-09-13T12:03:34.433 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42317.log 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38253.log 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42443.log 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42317.log.gz 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45215.log 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38253.log.gz 2024-09-13T12:03:34.434 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42443.log.gz 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45215.log: /var/log/ceph/ceph-mon.a.log 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45215.log.gz 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39773.log 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45005.log 2024-09-13T12:03:34.435 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39773.log.gz 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45467.log 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45005.log.gz 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2024-09-13T12:03:34.436 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45467.log.gz 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42989.log 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43429.log 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42611.log 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42989.log.gz 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43429.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43429.log.gz 2024-09-13T12:03:34.437 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42611.log.gz 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42065.log 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43115.log 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38260.log 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42065.log.gz 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42233.log 2024-09-13T12:03:34.438 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43115.log.gz 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38260.log.gz 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42149.log 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42233.log.gz 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42947.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-09-13T12:03:34.439 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41791.log 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42149.log.gz 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40214.log 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41596.log 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr: 25.8% -- replaced with /var/log/ceph/ceph-client.admin.41791.log.gz 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37687.log 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40214.log: /var/log/ceph/ceph-client.admin.41596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38232.log 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41596.log.gz 2024-09-13T12:03:34.440 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40214.log.gz 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37687.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37687.log.gz 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38232.log.gz 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41875.log 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41482.log 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2024-09-13T12:03:34.441 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45635.log 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41875.log.gz 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41482.log: /var/log/ceph/ceph-client.admin.45635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 18.5% -- replaced with /var/log/ceph/ceph-client.admin.45635.log.gz 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41482.log.gz 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2024-09-13T12:03:34.442 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43439.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.39835.log 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39835.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2024-09-13T12:03:34.443 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0%gzip -- replaced with /var/log/ceph/ceph.log.gz -5 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43073.log 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr: 26.2% -- replaced with /var/log/ceph/ceph-client.admin.43073.log.gz 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-09-13T12:03:34.444 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42695.log 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gz 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39958.log 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38904.log 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42695.log.gz 2024-09-13T12:03:34.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38257.log 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39958.log.gz 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5/var/log/ceph/ceph-client.admin.38904.log: --verbose -- /var/log/ceph/ceph-client.admin.33500.log 0.0% 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38904.log.gz 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38257.log.gz 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41012.log 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.33500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33500.log.gz 2024-09-13T12:03:34.446 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44107.log 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr: 17.5% -- replaced with /var/log/ceph/ceph-client.admin.41012.log.gz 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44753.log: /var/log/ceph/ceph-client.admin.44107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43031.log 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44107.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2024-09-13T12:03:34.447 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40855.log 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43031.log.gz 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44501.log 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40855.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40855.log.gz 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38244.log 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44501.log.gz 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-13T12:03:34.448 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38244.log.gz 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42359.log 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45089.log 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43421.log 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42359.log.gz 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39521.log 2024-09-13T12:03:34.449 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45089.log: /var/log/ceph/ceph-client.admin.43421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45089.log.gz 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43421.log.gz 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39521.log.gz 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43450.log: /var/log/ceph/ceph-client.admin.41555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43454.log 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2024-09-13T12:03:34.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38814.log 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33895.log 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43454.log: /var/log/ceph/ceph-client.admin.38814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43447.log 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43454.log.gz 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38814.log.gz 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.33895.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44543.log 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33895.log.gz 2024-09-13T12:03:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45257.log 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43447.log.gz 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44543.log: /var/log/ceph/ceph-client.admin.45257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38896.log 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44543.log.gz 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45257.log.gz 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38967.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.44627.log 2024-09-13T12:03:34.452 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45929.log 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38896.log: /var/log/ceph/ceph-client.admin.44627.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44627.log.gz 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.39731.log 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38896.log.gz 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45929.log.gz 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40515.log 2024-09-13T12:03:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46013.log 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.40515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40515.log.gz 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45887.log 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46013.log.gz 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45803.log 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45887.log.gz 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45551.log 2024-09-13T12:03:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45803.log.gz 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45845.log 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45551.log.gz 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45845.log.gz 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42821.log 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44113.log 2024-09-13T12:03:34.455 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42821.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37925.log 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42821.log.gz 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41440.log 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44113.log.gz 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39563.log 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37925.log: /var/log/ceph/ceph-client.admin.41440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41440.log.gz 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37925.log.gz 2024-09-13T12:03:34.456 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39563.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43957.log 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44098.log 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44669.log: /var/log/ceph/ceph-client.admin.43957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38870.log 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43957.log.gz 2024-09-13T12:03:34.457 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39957.log 2024-09-13T12:03:34.458 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44098.log.gz 2024-09-13T12:03:34.458 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.38870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38870.log.gz 2024-09-13T12:03:34.458 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.39957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39957.log.gz 2024-09-13T12:03:34.459 INFO:teuthology.orchestra.run.smithi037.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T12:03:34.459 INFO:teuthology.orchestra.run.smithi005.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T12:03:34.460 INFO:teuthology.orchestra.run.smithi005.stderr: 90.4% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-13T12:03:34.460 INFO:teuthology.orchestra.run.smithi087.stderr: 90.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T12:03:34.460 INFO:teuthology.orchestra.run.smithi087.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-13T12:03:34.460 INFO:teuthology.orchestra.run.smithi039.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T12:03:34.461 INFO:teuthology.orchestra.run.smithi039.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-13T12:03:34.539 INFO:teuthology.orchestra.run.smithi005.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T12:03:34.735 INFO:teuthology.orchestra.run.smithi087.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-13T12:03:34.743 INFO:teuthology.orchestra.run.smithi037.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T12:03:34.744 INFO:teuthology.orchestra.run.smithi037.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T12:03:34.747 INFO:teuthology.orchestra.run.smithi087.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-13T12:03:34.749 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-09-13T12:03:34.749 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m0.443s 2024-09-13T12:03:34.750 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m1.107s 2024-09-13T12:03:34.750 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.066s 2024-09-13T12:03:34.758 INFO:teuthology.orchestra.run.smithi037.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-13T12:03:34.764 INFO:teuthology.orchestra.run.smithi039.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T12:03:34.794 INFO:teuthology.orchestra.run.smithi037.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-13T12:03:34.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-09-13T12:03:34.796 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m0.494s 2024-09-13T12:03:34.796 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m1.830s 2024-09-13T12:03:34.796 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.082s 2024-09-13T12:03:34.832 INFO:teuthology.orchestra.run.smithi039.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-13T12:03:34.837 INFO:teuthology.orchestra.run.smithi005.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T12:03:34.863 INFO:teuthology.orchestra.run.smithi005.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-13T12:03:34.865 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-09-13T12:03:34.865 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m0.564s 2024-09-13T12:03:34.865 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m1.443s 2024-09-13T12:03:34.865 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.185s 2024-09-13T12:03:34.982 INFO:teuthology.orchestra.run.smithi039.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-13T12:03:34.984 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-13T12:03:34.984 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m0.681s 2024-09-13T12:03:34.985 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m1.843s 2024-09-13T12:03:34.985 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.071s 2024-09-13T12:03:34.985 INFO:tasks.ceph:Archiving logs... 2024-09-13T12:03:34.985 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi005/log 2024-09-13T12:03:34.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T12:03:35.380 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi037/log 2024-09-13T12:03:35.381 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T12:03:35.683 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi039/log 2024-09-13T12:03:35.684 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T12:03:35.972 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi087/log 2024-09-13T12:03:35.973 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T12:03:36.200 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T12:03:36.209 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T12:03:36.210 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T12:03:36.212 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T12:03:36.217 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T12:03:36.219 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T12:03:36.363 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T12:03:36.372 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T12:03:36.372 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:03:36.374 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:03:36.377 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:03:36.380 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T12:03:36.390 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:03:36.393 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:03:36.395 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:03:36.395 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-09-13T12:03:36.396 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 7 -75us[ -66us] +/- 65ms 2024-09-13T12:03:36.396 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 +679us[ +688us] +/- 63ms 2024-09-13T12:03:36.396 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -607us[ -599us] +/- 65ms 2024-09-13T12:03:36.396 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:03:36.397 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:03:36.397 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2024-09-13T12:03:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:03:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-09-13T12:03:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 5 -55us[ -55us] +/- 65ms 2024-09-13T12:03:36.399 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 +700us[ +707us] +/- 63ms 2024-09-13T12:03:36.400 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -605us[ -598us] +/- 65ms 2024-09-13T12:03:36.400 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:03:36.400 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:03:36.400 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-09-13T12:03:36.400 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 5 -70us[ -65us] +/- 65ms 2024-09-13T12:03:36.401 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 +694us[ +699us] +/- 63ms 2024-09-13T12:03:36.401 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -596us[ -596us] +/- 65ms 2024-09-13T12:03:36.401 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:03:36.402 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T12:03:36.402 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-09-13T12:03:36.402 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 4 -71us[ -71us] +/- 65ms 2024-09-13T12:03:36.403 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 +684us[ +690us] +/- 63ms 2024-09-13T12:03:36.403 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -602us[ -602us] +/- 65ms 2024-09-13T12:03:36.403 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T12:03:36.404 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T12:03:36.413 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T12:03:36.413 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T12:03:36.425 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T12:03:36.512 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T12:03:36.583 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T12:03:36.654 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T12:03:36.723 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T12:03:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:36.759 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T12:03:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:36.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T12:03:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:36.828 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T12:03:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:36.861 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T12:03:36.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T12:03:36.880 INFO:teuthology.task.internal:Duration was 571.504272 seconds 2024-09-13T12:03:36.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T12:03:36.889 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T12:03:36.889 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T12:03:36.892 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T12:03:36.895 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T12:03:36.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T12:03:36.935 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:03:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:03:36.941 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:03:36.942 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T12:03:37.346 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T12:03:37.346 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-09-13T12:03:37.346 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T12:03:37.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-09-13T12:03:37.372 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T12:03:37.406 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-13T12:03:37.406 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T12:03:37.438 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-09-13T12:03:37.438 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T12:03:37.469 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T12:03:37.469 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T12:03:37.472 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T12:03:37.474 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T12:03:37.481 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T12:03:37.542 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T12:03:37.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T12:03:37.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T12:03:37.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T12:03:37.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T12:03:37.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T12:03:37.695 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T12:03:37.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T12:03:37.721 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T12:03:37.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T12:03:37.776 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T12:03:37.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T12:03:37.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.815 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.817 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.819 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.837 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-09-13T12:03:37.840 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-09-13T12:03:37.842 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-13T12:03:37.865 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-09-13T12:03:37.898 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:37.949 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:37.964 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:37.980 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T12:03:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T12:03:37.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T12:03:38.014 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T12:03:38.015 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi005 2024-09-13T12:03:38.016 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T12:03:38.057 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi037 2024-09-13T12:03:38.057 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T12:03:38.092 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi039 2024-09-13T12:03:38.093 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T12:03:38.128 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903463/remote/smithi087 2024-09-13T12:03:38.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T12:03:38.162 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T12:03:38.163 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T12:03:38.165 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T12:03:38.167 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T12:03:38.170 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T12:03:38.213 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T12:03:38.225 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T12:03:38.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T12:03:38.234 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T12:03:38.234 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T12:03:38.236 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T12:03:38.238 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T12:03:38.241 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T12:03:38.249 INFO:teuthology.orchestra.run.smithi005.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 12:03 /home/ubuntu/cephtest 2024-09-13T12:03:38.252 INFO:teuthology.orchestra.run.smithi037.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 12:03 /home/ubuntu/cephtest 2024-09-13T12:03:38.254 INFO:teuthology.orchestra.run.smithi039.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 12:03 /home/ubuntu/cephtest 2024-09-13T12:03:38.269 INFO:teuthology.orchestra.run.smithi087.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 12:03 /home/ubuntu/cephtest 2024-09-13T12:03:38.271 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T12:03:38.281 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T12:03:38.305 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small} duration: 571.5042715072632 failure_reason: No module named 'tasks.rados' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c7986b92c3a04dadb7cdbc40388c9cfa status: fail success: false 2024-09-13T12:03:38.305 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T12:03:38.402 INFO:teuthology.run:FAIL