2024-04-22T01:47:11.648 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T01:47:11.649 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104 --name yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi --verbose -- /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/orig.config.yaml 2024-04-22T01:47:11.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:47:11.781 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104 branch: ci-yingxin-crimson-fix-tri-mutex-3 description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7668104' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-fix-tri-mutex-3 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: ci-yingxin-crimson-fix-tri-mutex-3 env: CRIMSON_COMPAT: '1' sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-fix-tri-mutex-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKktq2SK+j+hRsjA8azgEg0Ch7sfdBKBzBmFRSDqNg+A+9nIkMS2Pj/FeT2gIg7W6d0c/VDs8uku7iiOfbovN4= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6fG8tz02/G/GPRaj1/U74lFo8yZzXxkcBYrMbQh8/NNLzWU098PBo/Lux++g9b6vl3k1FG/SmkSl2fQFdkDrY= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 - radosbench: clients: - client.0 concurrency: 128 size: 8192 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_01:37:40 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T01:47:11.781 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa; will attempt to use it 2024-04-22T01:47:11.782 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks 2024-04-22T01:47:11.782 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T01:47:11.783 INFO:teuthology.task.internal:Checking packages... 2024-04-22T01:47:11.812 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '676947e1f73a9d7f900df5a6854c283c4eef24a9' 2024-04-22T01:47:11.812 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T01:47:11.812 INFO:teuthology.packaging:ref: None 2024-04-22T01:47:11.812 INFO:teuthology.packaging:tag: None 2024-04-22T01:47:11.812 INFO:teuthology.packaging:branch: ci-yingxin-crimson-fix-tri-mutex-3 2024-04-22T01:47:11.812 INFO:teuthology.packaging:sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:47:11.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-fix-tri-mutex-3 2024-04-22T01:47:11.955 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3159.g676947e1 2024-04-22T01:47:11.956 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T01:47:11.962 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T01:47:11.962 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T01:47:11.979 INFO:teuthology.task.internal:Saving configuration 2024-04-22T01:47:11.996 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T01:47:12.001 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T01:47:12.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:41:50.462463', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKktq2SK+j+hRsjA8azgEg0Ch7sfdBKBzBmFRSDqNg+A+9nIkMS2Pj/FeT2gIg7W6d0c/VDs8uku7iiOfbovN4='} 2024-04-22T01:47:12.043 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:41:50.460946', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6fG8tz02/G/GPRaj1/U74lFo8yZzXxkcBYrMbQh8/NNLzWU098PBo/Lux++g9b6vl3k1FG/SmkSl2fQFdkDrY='} 2024-04-22T01:47:12.043 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T01:47:12.049 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-04-22T01:47:12.049 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-22T01:47:12.050 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T01:47:12.126 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8dc11e7e50>, signals=[15]) 2024-04-22T01:47:12.127 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T01:47:12.133 INFO:teuthology.task.internal:Opening connections... 2024-04-22T01:47:12.133 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T01:47:12.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:47:12.216 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T01:47:12.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:47:12.290 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T01:47:12.297 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-22T01:47:12.314 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-22T01:47:12.314 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-22T01:47:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-22T01:47:12.373 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T01:47:12.373 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-22T01:47:12.373 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T01:47:12.373 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T01:47:12.373 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T01:47:12.373 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-22T01:47:12.393 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-04-22T01:47:12.411 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-04-22T01:47:12.411 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T01:47:12.468 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T01:47:12.469 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T01:47:12.469 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-04-22T01:47:12.493 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T01:47:12.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T01:47:12.507 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T01:47:12.507 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-04-22T01:47:12.510 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-04-22T01:47:12.525 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T01:47:12.531 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T01:47:12.531 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-04-22T01:47:12.568 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-04-22T01:47:12.676 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T01:47:12.727 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T01:47:12.742 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-22T01:47:12.742 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-22T01:47:12.742 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T01:47:12.743 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T01:47:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:47:12.759 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-04-22T01:47:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:47:12.760 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-04-22T01:47:12.816 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-437.el9.x86_64 2024-04-22T01:47:12.816 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-437.el9.x86_64 2024-04-22T01:47:12.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-04-22T01:47:12.817 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-437.el9.x86_64 2024-04-22T01:47:12.818 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-437.el9.x86_64 2024-04-22T01:47:12.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-04-22T01:47:16.676 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-04-22T01:47:16.842 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-04-22T01:47:20.852 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-04-22T01:47:25.019 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-22T01:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-22T01:47:28.171 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-22T01:47:33.254 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T01:47:34.242 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-04-22T01:47:35.151 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T01:47:35.152 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T01:47:35.238 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:47:35.239 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:47:35.239 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:47:35.346 DEBUG:teuthology.orchestra.run.smithi107:> echo no | sudo yum reinstall kernel || true 2024-04-22T01:47:35.875 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 01:47:33 AM UTC. 2024-04-22T01:47:36.001 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:47:36.008 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.009 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:36.009 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 255 k 2024-04-22T01:47:36.009 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2024-04-22T01:47:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:Operation aborted. 2024-04-22T01:47:36.067 INFO:teuthology.orchestra.run.smithi107.stdout:Is this ok [y/N]: 2024-04-22T01:47:36.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum reinstall -y kernel || true 2024-04-22T01:47:36.556 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 01:47:33 AM UTC. 2024-04-22T01:47:36.673 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalling: 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 255 k 2024-04-22T01:47:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 0 2024-04-22T01:47:36.675 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:47:37.180 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T01:47:37.180 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:47:37.180 INFO:teuthology.orchestra.run.smithi107.stdout:Total 503 kB/s | 255 kB 00:00 2024-04-22T01:47:37.199 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:47:37.364 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:47:37.364 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:47:37.369 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:47:37.369 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:47:37.501 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:47:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:47:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:38.304 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:47:38.800 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:Reinstalled: 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:47:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:47:39.200 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T01:47:39.246 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T01:47:39.247 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T01:47:39.247 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T01:47:39.247 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T01:47:39.247 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T01:47:39.286 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-04-22T01:47:39.306 DEBUG:teuthology.parallel:result is None 2024-04-22T01:47:40.518 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-22T01:47:41.428 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-04-22T01:47:46.778 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 6.4 kB/s | 1.7 kB 00:00 2024-04-22T01:47:48.701 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T01:47:48.701 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T01:47:48.793 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:47:48.794 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:47:48.794 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:47:48.918 DEBUG:teuthology.orchestra.run.smithi150:> echo no | sudo yum reinstall kernel || true 2024-04-22T01:47:49.420 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 01:47:46 AM UTC. 2024-04-22T01:47:49.548 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalling: 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:47:49.563 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 255 k 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 0 2024-04-22T01:47:49.564 INFO:teuthology.orchestra.run.smithi150.stderr:Operation aborted. 2024-04-22T01:47:49.617 INFO:teuthology.orchestra.run.smithi150.stdout:Is this ok [y/N]: 2024-04-22T01:47:49.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum reinstall -y kernel || true 2024-04-22T01:47:50.108 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 01:47:46 AM UTC. 2024-04-22T01:47:50.223 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalling: 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:47:50.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:50.225 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 255 k 2024-04-22T01:47:50.225 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 0 2024-04-22T01:47:50.225 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:47:51.380 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-437.el9.x86_64.rpm 245 kB/s | 255 kB 00:01 2024-04-22T01:47:51.380 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:47:51.381 INFO:teuthology.orchestra.run.smithi150.stdout:Total 220 kB/s | 255 kB 00:01 2024-04-22T01:47:51.403 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:47:51.587 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:47:51.588 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:47:51.592 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:47:51.593 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:47:51.722 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:47:51.789 INFO:teuthology.orchestra.run.smithi150.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:47:51.790 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:52.472 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:52.472 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:47:53.171 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:47:53.171 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:53.171 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalled: 2024-04-22T01:47:53.172 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T01:47:53.172 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:47:53.172 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:47:53.341 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T01:47:53.388 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T01:47:53.388 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T01:47:53.388 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T01:47:53.389 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-22T01:47:53.389 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T01:47:53.416 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-04-22T01:47:53.435 DEBUG:teuthology.parallel:result is None 2024-04-22T01:47:53.435 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T01:47:53.442 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T01:47:53.443 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T01:47:53.445 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T01:47:53.490 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T01:47:53.497 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T01:47:53.507 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T01:47:53.507 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T01:47:53.510 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T01:47:53.570 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T01:47:53.577 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T01:47:53.577 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T01:47:53.580 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T01:47:53.610 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:47:53.619 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:47:53.621 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:47:53.628 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:47:53.630 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T01:47:53.637 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T01:47:53.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T01:47:53.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T01:47:53.693 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T01:47:53.703 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T01:47:53.703 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T01:47:53.732 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T01:47:53.747 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:47:53.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:47:53.876 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:47:53.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:47:54.000 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:47:54.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T01:47:54.067 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:47:54.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:47:54.159 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:47:54.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:47:54.279 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:47:54.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T01:47:54.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-04-22T01:47:54.346 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-04-22T01:47:54.387 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:47:54.420 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:47:54.726 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T01:47:54.734 INFO:teuthology.task.internal:Starting timer... 2024-04-22T01:47:54.734 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T01:47:54.744 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T01:47:54.752 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-22T01:47:54.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-04-22T01:47:54.823 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-04-22T01:47:54.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-04-22T01:47:54.878 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-04-22T01:47:54.880 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T01:47:54.880 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-04-22T01:47:54.902 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-04-22T01:47:54.903 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-04-22T01:47:54.927 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-04-22T01:47:54.928 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi150.front.sepia.ceph.com': 'permissive'} 2024-04-22T01:47:54.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:47:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:47:54.969 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:47:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:47:54.994 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T01:47:54.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-04-22T01:47:55.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-04-22T01:47:55.057 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T01:47:55.069 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T01:47:55.084 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T01:47:55.085 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T01:51:23.767 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-04-22T01:51:23.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-22T01:51:23.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:51:23.846 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-22T01:51:23.921 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-22T01:51:23.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-22T01:51:23.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:51:23.994 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-04-22T01:51:24.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-22T01:51:24.073 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T01:51:24.083 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T01:51:24.083 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T01:51:24.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:51:24.086 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T01:51:24.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:51:24.111 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T01:51:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T01:51:24.138 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T01:51:24.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-04-22T01:51:24.151 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T01:51:24.157 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-04-22T01:51:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T01:51:24.174 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-04-22T01:51:24.183 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T01:51:24.184 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-04-22T01:51:24.197 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T01:51:24.210 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T01:51:24.236 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.240 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.262 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:51:24.266 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:51:24.266 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-04-22T01:51:24.266 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.267 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.267 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.267 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:51:24.267 INFO:teuthology.run_tasks:Running task install... 2024-04-22T01:51:24.275 DEBUG:teuthology.task.install:project ceph 2024-04-22T01:51:24.275 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}} 2024-04-22T01:51:24.275 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'} 2024-04-22T01:51:24.276 INFO:teuthology.task.install:Using flavor: crimson 2024-04-22T01:51:24.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T01:51:24.281 INFO:teuthology.task.install:extra packages: [] 2024-04-22T01:51:24.282 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'tag': None, 'wait_for_package': False} 2024-04-22T01:51:24.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:51:24.284 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'tag': None, 'wait_for_package': False} 2024-04-22T01:51:24.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:51:24.430 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/ 2024-04-22T01:51:24.430 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3159.g676947e1 2024-04-22T01:51:24.433 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/ 2024-04-22T01:51:24.433 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3159.g676947e1 2024-04-22T01:51:24.552 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T01:51:24.553 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:51:24.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T01:51:24.577 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T01:51:24.577 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/676947e1f73a9d7f900df5a6854c283c4eef24a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T01:51:24.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T01:51:24.699 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T01:51:24.700 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:51:24.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T01:51:24.715 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T01:51:24.724 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T01:51:24.725 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/676947e1f73a9d7f900df5a6854c283c4eef24a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T01:51:24.780 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-04-22T01:51:24.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-04-22T01:51:24.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T01:51:24.863 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T01:51:24.926 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-04-22T01:51:24.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-04-22T01:51:25.007 INFO:teuthology.orchestra.run.smithi150.stdout:48 files removed 2024-04-22T01:51:25.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw 2024-04-22T01:51:25.151 INFO:teuthology.orchestra.run.smithi107.stdout:48 files removed 2024-04-22T01:51:25.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-04-22T01:51:25.684 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 224 kB/s | 87 kB 00:00 2024-04-22T01:51:25.827 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 222 kB/s | 87 kB 00:00 2024-04-22T01:51:26.126 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-22T01:51:26.277 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-22T01:51:26.543 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 5.3 kB/s | 2.0 kB 00:00 2024-04-22T01:51:26.794 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 4.2 kB/s | 2.0 kB 00:00 2024-04-22T01:51:27.944 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 8.0 MB 00:01 2024-04-22T01:51:29.978 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-22T01:51:34.412 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.2 MB 00:00 2024-04-22T01:51:36.088 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-22T01:51:36.318 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 868 kB/s | 8.0 MB 00:09 2024-04-22T01:51:38.240 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-22T01:51:43.240 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T01:51:45.456 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:51:45.459 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:51:45.459 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:51:45.459 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:51:45.459 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:51:45.459 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 10 M 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.7 M 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T01:51:45.460 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.9 M 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 19 M 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-3159.g676947e1.el9 ceph 25 k 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 796 k 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T01:51:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T01:51:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T01:51:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 549 k 2024-04-22T01:51:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T01:51:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.3 M 2024-04-22T01:51:45.462 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 45 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 135 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-3159.g676947e1.el9 ceph 160 k 2024-04-22T01:51:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-3159.g676947e1.el9 ceph 315 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 293 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 99 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T01:51:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:51:45.465 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:51:45.466 INFO:teuthology.orchestra.run.smithi150.stdout:Install 30 Packages 2024-04-22T01:51:45.466 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 2 Packages 2024-04-22T01:51:45.466 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:51:45.467 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 59 M 2024-04-22T01:51:45.467 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:51:47.216 INFO:teuthology.orchestra.run.smithi150.stdout:(1/32): ceph-base-19.0.0-3159.g676947e1.el9.x86 4.1 MB/s | 5.9 MB 00:01 2024-04-22T01:51:47.307 INFO:teuthology.orchestra.run.smithi150.stdout:(2/32): ceph-selinux-19.0.0-3159.g676947e1.el9. 275 kB/s | 25 kB 00:00 2024-04-22T01:51:47.583 INFO:teuthology.orchestra.run.smithi150.stdout:(3/32): libcephfs2-19.0.0-3159.g676947e1.el9.x8 2.8 MB/s | 796 kB 00:00 2024-04-22T01:51:47.800 INFO:teuthology.orchestra.run.smithi150.stdout:(4/32): ceph-common-19.0.0-3159.g676947e1.el9.x 9.6 MB/s | 19 MB 00:02 2024-04-22T01:51:47.841 INFO:teuthology.orchestra.run.smithi150.stdout:(5/32): libradosstriper1-19.0.0-3159.g676947e1. 2.1 MB/s | 549 kB 00:00 2024-04-22T01:51:47.925 INFO:teuthology.orchestra.run.smithi150.stdout:(6/32): python3-ceph-argparse-19.0.0-3159.g6769 541 kB/s | 45 kB 00:00 2024-04-22T01:51:48.017 INFO:teuthology.orchestra.run.smithi150.stdout:(7/32): python3-ceph-common-19.0.0-3159.g676947 1.4 MB/s | 135 kB 00:00 2024-04-22T01:51:48.109 INFO:teuthology.orchestra.run.smithi150.stdout:(8/32): python3-cephfs-19.0.0-3159.g676947e1.el 1.7 MB/s | 160 kB 00:00 2024-04-22T01:51:48.268 INFO:teuthology.orchestra.run.smithi150.stdout:(9/32): python3-rados-19.0.0-3159.g676947e1.el9 1.9 MB/s | 315 kB 00:00 2024-04-22T01:51:48.385 INFO:teuthology.orchestra.run.smithi150.stdout:(10/32): librgw2-19.0.0-3159.g676947e1.el9.x86_ 9.0 MB/s | 5.3 MB 00:00 2024-04-22T01:51:48.418 INFO:teuthology.orchestra.run.smithi150.stdout:(11/32): python3-rbd-19.0.0-3159.g676947e1.el9. 1.9 MB/s | 293 kB 00:00 2024-04-22T01:51:48.443 INFO:teuthology.orchestra.run.smithi150.stdout:(12/32): python3-rgw-19.0.0-3159.g676947e1.el9. 1.7 MB/s | 99 kB 00:00 2024-04-22T01:51:48.870 INFO:teuthology.orchestra.run.smithi150.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-04-22T01:51:48.995 INFO:teuthology.orchestra.run.smithi150.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2024-04-22T01:51:49.020 INFO:teuthology.orchestra.run.smithi150.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 278 kB/s | 160 kB 00:00 2024-04-22T01:51:49.246 INFO:teuthology.orchestra.run.smithi150.stdout:(16/32): ceph-radosgw-19.0.0-3159.g676947e1.el9 2.9 MB/s | 10 MB 00:03 2024-04-22T01:51:49.270 INFO:teuthology.orchestra.run.smithi150.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-04-22T01:51:49.379 INFO:teuthology.orchestra.run.smithi150.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-04-22T01:51:49.455 INFO:teuthology.orchestra.run.smithi150.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-04-22T01:51:49.479 INFO:teuthology.orchestra.run.smithi150.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 358 kB/s | 36 kB 00:00 2024-04-22T01:51:49.555 INFO:teuthology.orchestra.run.smithi150.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-04-22T01:51:49.588 INFO:teuthology.orchestra.run.smithi150.stdout:(22/32): python3-prettytable-0.7.2-27.el9.noarc 123 kB/s | 42 kB 00:00 2024-04-22T01:51:49.664 INFO:teuthology.orchestra.run.smithi150.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-04-22T01:51:49.689 INFO:teuthology.orchestra.run.smithi150.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-22T01:51:49.714 INFO:teuthology.orchestra.run.smithi150.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-04-22T01:51:49.773 INFO:teuthology.orchestra.run.smithi150.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-04-22T01:51:49.815 INFO:teuthology.orchestra.run.smithi150.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-22T01:51:49.865 INFO:teuthology.orchestra.run.smithi150.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-22T01:51:49.915 INFO:teuthology.orchestra.run.smithi150.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-22T01:51:50.241 INFO:teuthology.orchestra.run.smithi150.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-04-22T01:51:50.416 INFO:teuthology.orchestra.run.smithi150.stdout:(31/32): librbd1-19.0.0-3159.g676947e1.el9.x86_ 7.3 MB/s | 3.7 MB 00:00 2024-04-22T01:51:50.749 INFO:teuthology.orchestra.run.smithi150.stdout:(32/32): librados2-19.0.0-3159.g676947e1.el9.x8 3.8 MB/s | 3.4 MB 00:00 2024-04-22T01:51:50.751 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:51:50.752 INFO:teuthology.orchestra.run.smithi150.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-22T01:51:50.856 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:51:50.893 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:51:50.893 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:51:51.352 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:51:51.352 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:51:52.828 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:51:53.295 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T01:51:53.402 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:51:53.492 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:51:53.581 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:51:53.612 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:51:53.674 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:51:53.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T01:51:53.741 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T01:51:53.766 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T01:51:53.789 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:51:53.811 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:51:54.005 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:51:54.038 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:51:54.387 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T01:51:54.434 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:51:54.501 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:51:54.551 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 12/34 2024-04-22T01:51:54.606 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:51:54.631 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 14/34 2024-04-22T01:51:54.709 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T01:51:54.737 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:51:54.823 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:51:54.852 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T01:51:54.960 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T01:51:54.994 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T01:51:55.034 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T01:51:55.060 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T01:51:55.088 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T01:51:55.119 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T01:51:55.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T01:51:55.722 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T01:51:55.751 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:51:55.824 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:51:55.901 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 27/34 2024-04-22T01:51:55.930 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 28/34 2024-04-22T01:51:58.378 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:51:58.410 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:51:58.857 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 907 kB/s | 19 MB 00:21 2024-04-22T01:51:59.077 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:51:59.114 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:51:59.202 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:51:59.202 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T01:51:59.202 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T01:51:59.202 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:51:59.248 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:03.982 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-22T01:52:05.483 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-04-22T01:52:07.134 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-22T01:52:08.999 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:08.999 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-04-22T01:52:08.999 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-04-22T01:52:08.999 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-04-22T01:52:08.999 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-04-22T01:52:09.000 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-04-22T01:52:09.000 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-04-22T01:52:09.000 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-04-22T01:52:09.000 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:10.039 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:52:10.063 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:52:10.064 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:52:10.064 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:52:10.064 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:52:10.064 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:52:10.064 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:10.065 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:52:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:52:10.149 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:11.486 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 1/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 5/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 6/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 7/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 8/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 9/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 10/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:52:11.487 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 12/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T01:52:11.488 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T01:52:11.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 33/34 2024-04-22T01:52:12.217 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:52:12.263 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:52:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:52:12.265 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:52:12.266 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:52:12.266 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:12.266 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:52:12.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-test 2024-04-22T01:52:13.105 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:30 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:52:13.231 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-3159.g676947e1.el9 ceph 84 M 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-3159.g676947e1.el9 ceph 174 k 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T01:52:13.233 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout:Install 9 Packages 2024-04-22T01:52:13.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:13.235 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 87 M 2024-04-22T01:52:13.235 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 812 M 2024-04-22T01:52:13.235 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:52:14.037 INFO:teuthology.orchestra.run.smithi150.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 239 kB/s | 94 kB 00:00 2024-04-22T01:52:14.079 INFO:teuthology.orchestra.run.smithi150.stdout:(2/9): libcephsqlite-19.0.0-3159.g676947e1.el9. 399 kB/s | 174 kB 00:00 2024-04-22T01:52:14.383 INFO:teuthology.orchestra.run.smithi150.stdout:(3/9): libubsan-11.4.1-3.el9.x86_64.rpm 617 kB/s | 187 kB 00:00 2024-04-22T01:52:14.409 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:52:14.411 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:14.411 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 10 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.7 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.9 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 19 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-3159.g676947e1.el9 ceph 25 k 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T01:52:14.412 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 796 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 549 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.3 M 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 45 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 135 k 2024-04-22T01:52:14.413 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-3159.g676947e1.el9 ceph 160 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-3159.g676947e1.el9 ceph 315 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 293 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 99 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:14.414 INFO:teuthology.orchestra.run.smithi107.stdout:Install 30 Packages 2024-04-22T01:52:14.415 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-04-22T01:52:14.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:14.415 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 59 M 2024-04-22T01:52:14.415 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:52:14.467 INFO:teuthology.orchestra.run.smithi150.stdout:(4/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.1 MB/s | 469 kB 00:00 2024-04-22T01:52:14.634 INFO:teuthology.orchestra.run.smithi150.stdout:(5/9): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-04-22T01:52:14.693 INFO:teuthology.orchestra.run.smithi150.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T01:52:14.861 INFO:teuthology.orchestra.run.smithi150.stdout:(7/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.1 MB/s | 1.0 MB 00:00 2024-04-22T01:52:15.295 INFO:teuthology.orchestra.run.smithi150.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 202 kB/s | 122 kB 00:00 2024-04-22T01:52:18.892 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-base-19.0.0-3159.g676947e1.el9.x86 1.5 MB/s | 5.9 MB 00:03 2024-04-22T01:52:18.959 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-selinux-19.0.0-3159.g676947e1.el9. 378 kB/s | 25 kB 00:00 2024-04-22T01:52:19.334 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): ceph-common-19.0.0-3159.g676947e1.el9.x 4.4 MB/s | 19 MB 00:04 2024-04-22T01:52:19.385 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): libcephfs2-19.0.0-3159.g676947e1.el9.x8 1.8 MB/s | 796 kB 00:00 2024-04-22T01:52:19.435 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): libradosstriper1-19.0.0-3159.g676947e1. 5.3 MB/s | 549 kB 00:00 2024-04-22T01:52:19.485 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): python3-ceph-argparse-19.0.0-3159.g6769 906 kB/s | 45 kB 00:00 2024-04-22T01:52:19.544 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): python3-ceph-common-19.0.0-3159.g676947 2.3 MB/s | 135 kB 00:00 2024-04-22T01:52:19.603 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-cephfs-19.0.0-3159.g676947e1.el 2.7 MB/s | 160 kB 00:00 2024-04-22T01:52:19.669 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-rados-19.0.0-3159.g676947e1.el9 4.6 MB/s | 315 kB 00:00 2024-04-22T01:52:19.736 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-rbd-19.0.0-3159.g676947e1.el9. 4.3 MB/s | 293 kB 00:00 2024-04-22T01:52:19.795 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-rgw-19.0.0-3159.g676947e1.el9. 1.7 MB/s | 99 kB 00:00 2024-04-22T01:52:19.978 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): ceph-radosgw-19.0.0-3159.g676947e1.el9 2.0 MB/s | 10 MB 00:05 2024-04-22T01:52:20.054 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 413 kB/s | 107 kB 00:00 2024-04-22T01:52:20.162 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 873 kB/s | 160 kB 00:00 2024-04-22T01:52:20.179 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2024-04-22T01:52:20.280 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-04-22T01:52:20.321 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-04-22T01:52:20.338 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 723 kB/s | 42 kB 00:00 2024-04-22T01:52:20.380 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-22T01:52:20.397 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-04-22T01:52:20.431 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-22T01:52:20.706 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): librgw2-19.0.0-3159.g676947e1.el9.x86_ 4.0 MB/s | 5.3 MB 00:01 2024-04-22T01:52:20.732 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 926 kB/s | 309 kB 00:00 2024-04-22T01:52:20.824 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 745 kB/s | 49 kB 00:00 2024-04-22T01:52:20.857 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 172 kB/s | 26 kB 00:00 2024-04-22T01:52:20.891 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-22T01:52:21.041 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-22T01:52:21.392 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-04-22T01:52:21.950 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): librados2-19.0.0-3159.g676947e1.el9.x8 6.1 MB/s | 3.4 MB 00:00 2024-04-22T01:52:22.109 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-04-22T01:52:22.592 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): librbd1-19.0.0-3159.g676947e1.el9.x86_ 5.7 MB/s | 3.7 MB 00:00 2024-04-22T01:52:22.892 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-04-22T01:52:22.898 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:52:22.898 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.0 MB/s | 59 MB 00:08 2024-04-22T01:52:22.995 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:52:23.031 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:52:23.032 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:52:23.337 INFO:teuthology.orchestra.run.smithi150.stdout:(9/9): ceph-test-19.0.0-3159.g676947e1.el9.x86_ 8.7 MB/s | 84 MB 00:09 2024-04-22T01:52:23.338 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:52:23.339 INFO:teuthology.orchestra.run.smithi150.stdout:Total 8.6 MB/s | 87 MB 00:10 2024-04-22T01:52:23.421 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:52:23.444 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:52:23.444 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:52:23.495 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:52:23.495 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:52:24.044 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:52:24.044 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:52:24.922 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:52:24.985 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T01:52:25.029 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T01:52:25.038 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:52:25.074 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T01:52:25.101 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T01:52:25.131 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:52:25.164 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:52:25.233 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:52:25.273 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T01:52:25.300 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:52:25.556 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T01:52:25.603 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:52:25.694 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:52:25.791 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:52:25.821 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:52:25.891 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:52:25.929 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T01:52:25.958 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T01:52:25.984 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T01:52:26.006 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:52:26.027 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:52:26.223 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:52:26.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:52:26.605 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T01:52:26.661 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:52:26.727 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:52:26.769 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 12/34 2024-04-22T01:52:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:52:26.849 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 14/34 2024-04-22T01:52:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T01:52:26.955 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:52:27.041 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:52:27.069 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T01:52:27.177 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T01:52:27.212 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T01:52:27.252 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T01:52:27.277 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T01:52:27.306 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T01:52:27.337 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T01:52:27.373 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T01:52:27.941 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T01:52:27.969 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:52:28.040 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:52:28.117 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 27/34 2024-04-22T01:52:28.154 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 28/34 2024-04-22T01:52:30.591 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:52:30.628 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:52:31.289 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:52:31.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:52:31.423 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:52:31.423 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T01:52:31.423 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T01:52:31.423 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:31.474 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:36.572 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:52:36.830 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-04-22T01:52:41.253 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:42.305 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:52:42.351 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:42.353 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:52:42.432 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:52:42.432 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:42.963 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:52:42.963 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 1/9 2024-04-22T01:52:42.963 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2/9 2024-04-22T01:52:42.963 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T01:52:42.964 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T01:52:42.964 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:52:42.964 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T01:52:42.964 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T01:52:42.964 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T01:52:43.293 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:43.294 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:52:43.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph 2024-04-22T01:52:43.748 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 1/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 5/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 6/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 7/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 8/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 9/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 10/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 12/34 2024-04-22T01:52:43.749 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T01:52:43.750 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T01:52:43.751 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 33/34 2024-04-22T01:52:44.089 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:01 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:52:44.211 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:52:44.213 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:44.213 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-3159.g676947e1.el9 ceph 6.5 k 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-3159.g676947e1.el9 ceph 2.2 M 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-3159.g676947e1.el9 ceph 1.6 M 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 249 k 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.0 M 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 71 M 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T01:52:44.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T01:52:44.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T01:52:44.216 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:Install 36 Packages 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 84 M 2024-04-22T01:52:44.217 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 922 M 2024-04-22T01:52:44.218 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:52:44.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:52:44.521 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.522 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:44.523 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:52:44.753 INFO:teuthology.orchestra.run.smithi150.stdout:(1/36): ceph-19.0.0-3159.g676947e1.el9.x86_64.r 31 kB/s | 6.5 kB 00:00 2024-04-22T01:52:44.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-04-22T01:52:44.887 INFO:teuthology.orchestra.run.smithi150.stdout:(2/36): ceph-mgr-19.0.0-3159.g676947e1.el9.x86_ 4.5 MB/s | 1.6 MB 00:00 2024-04-22T01:52:45.187 INFO:teuthology.orchestra.run.smithi150.stdout:(3/36): ceph-mds-19.0.0-3159.g676947e1.el9.x86_ 3.4 MB/s | 2.2 MB 00:00 2024-04-22T01:52:45.295 INFO:teuthology.orchestra.run.smithi150.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3159.g6769 2.3 MB/s | 249 kB 00:00 2024-04-22T01:52:45.366 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:33 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:52:45.490 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:52:45.490 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:45.490 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-3159.g676947e1.el9 ceph 84 M 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-3159.g676947e1.el9 ceph 174 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:45.491 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout:Install 9 Packages 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 87 M 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 812 M 2024-04-22T01:52:45.492 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:52:45.917 INFO:teuthology.orchestra.run.smithi107.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 605 kB/s | 94 kB 00:00 2024-04-22T01:52:46.027 INFO:teuthology.orchestra.run.smithi107.stdout:(2/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 4.2 MB/s | 469 kB 00:00 2024-04-22T01:52:46.085 INFO:teuthology.orchestra.run.smithi107.stdout:(3/9): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-04-22T01:52:46.178 INFO:teuthology.orchestra.run.smithi107.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-22T01:52:46.228 INFO:teuthology.orchestra.run.smithi107.stdout:(5/9): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-22T01:52:46.246 INFO:teuthology.orchestra.run.smithi150.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 267 kB/s | 253 kB 00:00 2024-04-22T01:52:46.287 INFO:teuthology.orchestra.run.smithi107.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-22T01:52:46.321 INFO:teuthology.orchestra.run.smithi107.stdout:(7/9): libcephsqlite-19.0.0-3159.g676947e1.el9. 310 kB/s | 174 kB 00:00 2024-04-22T01:52:46.447 INFO:teuthology.orchestra.run.smithi107.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 763 kB/s | 122 kB 00:00 2024-04-22T01:52:47.211 INFO:teuthology.orchestra.run.smithi150.stdout:(6/36): ceph-mon-19.0.0-3159.g676947e1.el9.x86_ 2.0 MB/s | 5.0 MB 00:02 2024-04-22T01:52:47.955 INFO:teuthology.orchestra.run.smithi150.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 143 kB/s | 106 kB 00:00 2024-04-22T01:52:48.755 INFO:teuthology.orchestra.run.smithi150.stdout:(8/36): ceph-osd-19.0.0-3159.g676947e1.el9.x86_ 18 MB/s | 71 MB 00:03 2024-04-22T01:52:48.822 INFO:teuthology.orchestra.run.smithi150.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 487 kB/s | 1.2 MB 00:02 2024-04-22T01:52:49.330 INFO:teuthology.orchestra.run.smithi150.stdout:(10/36): python3-pycparser-2.20-6.el9.noarch.rp 98 kB/s | 135 kB 00:01 2024-04-22T01:52:49.481 INFO:teuthology.orchestra.run.smithi150.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-22T01:52:49.531 INFO:teuthology.orchestra.run.smithi150.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-22T01:52:49.573 INFO:teuthology.orchestra.run.smithi150.stdout:(13/36): python3-requests-2.25.1-8.el9.noarch.r 158 kB/s | 125 kB 00:00 2024-04-22T01:52:49.598 INFO:teuthology.orchestra.run.smithi150.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 528 kB/s | 35 kB 00:00 2024-04-22T01:52:49.649 INFO:teuthology.orchestra.run.smithi150.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-22T01:52:49.682 INFO:teuthology.orchestra.run.smithi150.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 471 kB/s | 51 kB 00:00 2024-04-22T01:52:49.708 INFO:teuthology.orchestra.run.smithi150.stdout:(17/36): python3-urllib3-1.26.5-5.el9.noarch.rp 242 kB/s | 215 kB 00:00 2024-04-22T01:52:49.891 INFO:teuthology.orchestra.run.smithi150.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 278 kB/s | 43 kB 00:00 2024-04-22T01:52:49.933 INFO:teuthology.orchestra.run.smithi150.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 602 kB/s | 151 kB 00:00 2024-04-22T01:52:49.967 INFO:teuthology.orchestra.run.smithi150.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-22T01:52:50.017 INFO:teuthology.orchestra.run.smithi150.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-04-22T01:52:50.034 INFO:teuthology.orchestra.run.smithi150.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-04-22T01:52:50.076 INFO:teuthology.orchestra.run.smithi150.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-04-22T01:52:50.101 INFO:teuthology.orchestra.run.smithi150.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-04-22T01:52:50.135 INFO:teuthology.orchestra.run.smithi150.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-04-22T01:52:50.170 INFO:teuthology.orchestra.run.smithi150.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-04-22T01:52:50.185 INFO:teuthology.orchestra.run.smithi150.stdout:(27/36): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-04-22T01:52:50.202 INFO:teuthology.orchestra.run.smithi150.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-04-22T01:52:50.236 INFO:teuthology.orchestra.run.smithi150.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-04-22T01:52:50.278 INFO:teuthology.orchestra.run.smithi150.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-22T01:52:50.295 INFO:teuthology.orchestra.run.smithi150.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-04-22T01:52:50.345 INFO:teuthology.orchestra.run.smithi150.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 826 kB/s | 90 kB 00:00 2024-04-22T01:52:50.362 INFO:teuthology.orchestra.run.smithi150.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-04-22T01:52:50.387 INFO:teuthology.orchestra.run.smithi150.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-22T01:52:50.421 INFO:teuthology.orchestra.run.smithi150.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-04-22T01:52:50.471 INFO:teuthology.orchestra.run.smithi150.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 242 kB/s | 20 kB 00:00 2024-04-22T01:52:50.476 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:52:50.476 INFO:teuthology.orchestra.run.smithi150.stdout:Total 13 MB/s | 84 MB 00:06 2024-04-22T01:52:50.613 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:52:50.655 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:52:50.655 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:52:51.270 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:52:51.271 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:52:52.850 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:52:52.924 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T01:52:52.957 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T01:52:52.984 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T01:52:53.056 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T01:52:53.100 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T01:52:53.133 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T01:52:53.226 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T01:52:53.304 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T01:52:53.344 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T01:52:53.379 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T01:52:53.409 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T01:52:53.443 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T01:52:53.485 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T01:52:53.526 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T01:52:53.569 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T01:52:53.603 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T01:52:53.642 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T01:52:53.716 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T01:52:53.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T01:52:53.951 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T01:52:54.011 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T01:52:54.058 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T01:52:54.115 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T01:52:54.170 INFO:teuthology.orchestra.run.smithi107.stdout:(9/9): ceph-test-19.0.0-3159.g676947e1.el9.x86_ 10 MB/s | 84 MB 00:08 2024-04-22T01:52:54.171 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T01:52:54.172 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:52:54.172 INFO:teuthology.orchestra.run.smithi107.stdout:Total 10 MB/s | 87 MB 00:08 2024-04-22T01:52:54.251 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:52:54.273 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:52:54.273 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:52:54.359 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T01:52:54.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T01:52:54.501 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T01:52:54.597 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T01:52:54.637 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T01:52:54.756 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T01:52:54.892 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:52:54.893 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:52:54.946 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 31/36 2024-04-22T01:52:54.972 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:52:54.998 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:52:55.778 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:52:55.852 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T01:52:55.896 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T01:52:55.941 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T01:52:55.968 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T01:52:55.998 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:52:56.024 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:52:56.084 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:52:56.124 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T01:52:56.151 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:53:05.822 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:53:05.848 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:53:05.849 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:05.849 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T01:53:05.849 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:53:05.849 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:53:05.849 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:06.390 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:53:06.413 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:06.649 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:53:06.671 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:53:06.671 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:06.671 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T01:53:06.672 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:53:06.672 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:53:06.672 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:06.743 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:53:07.472 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:53:07.546 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 1/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T01:53:13.602 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:53:13.603 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T01:53:13.603 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T01:53:13.603 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 1/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 3/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 4/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 5/36 2024-04-22T01:53:13.628 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 6/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T01:53:13.629 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T01:53:13.630 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T01:53:13.631 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T01:53:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T01:53:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:13.967 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:53:13.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:13.968 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:14.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-04-22T01:53:14.577 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T01:53:14.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:14.578 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:53:14.579 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:53:14.580 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:53:14.581 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:53:14.582 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:14.582 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:14.765 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:02 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:14.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-base 2024-04-22T01:53:14.884 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-3159.g676947e1.el9 ceph 6.5 k 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:53:14.887 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-3159.g676947e1.el9 ceph 2.2 M 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-3159.g676947e1.el9 ceph 1.6 M 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 249 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.0 M 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 71 M 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T01:53:14.888 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T01:53:14.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2024-04-22T01:53:14.890 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 84 M 2024-04-22T01:53:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 922 M 2024-04-22T01:53:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:53:15.376 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:15.454 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:53:15.496 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:15.497 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:53:15.497 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:15.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install cephadm 2024-04-22T01:53:15.682 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-19.0.0-3159.g676947e1.el9.x86_64.r 27 kB/s | 6.5 kB 00:00 2024-04-22T01:53:15.790 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mgr-19.0.0-3159.g676947e1.el9.x86_ 4.4 MB/s | 1.6 MB 00:00 2024-04-22T01:53:15.865 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mds-19.0.0-3159.g676947e1.el9.x86_ 5.1 MB/s | 2.2 MB 00:00 2024-04-22T01:53:15.941 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3159.g6769 3.2 MB/s | 249 kB 00:00 2024-04-22T01:53:16.067 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:16.190 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:16.190 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 762 k 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 762 k 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 769 k 2024-04-22T01:53:16.191 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:16.949 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 251 kB/s | 253 kB 00:01 2024-04-22T01:53:17.107 INFO:teuthology.orchestra.run.smithi150.stdout:cephadm-19.0.0-3159.g676947e1.el9.noarch.rpm 832 kB/s | 762 kB 00:00 2024-04-22T01:53:17.108 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:17.108 INFO:teuthology.orchestra.run.smithi150.stdout:Total 830 kB/s | 762 kB 00:00 2024-04-22T01:53:17.108 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:17.113 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:17.113 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:17.120 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:17.120 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:17.190 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:17.613 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:17.703 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:18.033 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): ceph-mon-19.0.0-3159.g676947e1.el9.x86_ 2.1 MB/s | 5.0 MB 00:02 2024-04-22T01:53:18.061 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:18.309 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:18.310 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:18.310 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:18.310 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:18.310 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:18.310 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:18.570 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T01:53:19.083 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): ceph-osd-19.0.0-3159.g676947e1.el9.x86_ 22 MB/s | 71 MB 00:03 2024-04-22T01:53:19.094 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:19.200 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 91 kB/s | 106 kB 00:01 2024-04-22T01:53:19.218 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3159.g676947e1.el9 ceph 157 k 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:19.219 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:53:19.220 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:19.220 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 157 k 2024-04-22T01:53:19.220 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 640 k 2024-04-22T01:53:19.220 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:19.317 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): python3-cryptography-36.0.1-4.el9.x86_6 530 kB/s | 1.2 MB 00:02 2024-04-22T01:53:19.580 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-immutable-object-cache-19.0.0-3159.g676947 436 kB/s | 157 kB 00:00 2024-04-22T01:53:19.581 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:19.581 INFO:teuthology.orchestra.run.smithi150.stdout:Total 434 kB/s | 157 kB 00:00 2024-04-22T01:53:19.581 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:19.591 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:19.591 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:19.649 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:19.649 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:19.804 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:19.890 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:19.914 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:19.914 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:19.914 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:53:19.914 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:19.950 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-pycparser-2.20-6.el9.noarch.rp 156 kB/s | 135 kB 00:00 2024-04-22T01:53:19.992 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 158 kB/s | 125 kB 00:00 2024-04-22T01:53:20.317 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 220 kB/s | 215 kB 00:00 2024-04-22T01:53:20.783 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:20.783 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:20.783 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:20.784 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:20.784 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:20.784 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:20.801 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 222 kB/s | 188 kB 00:00 2024-04-22T01:53:20.826 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-04-22T01:53:20.918 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 58 kB/s | 35 kB 00:00 2024-04-22T01:53:21.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr 2024-04-22T01:53:21.068 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-04-22T01:53:21.335 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 626 kB/s | 151 kB 00:00 2024-04-22T01:53:21.411 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-04-22T01:53:21.523 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:21.544 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-pytz-2021.1-5.el9.noarch.rpm 69 kB/s | 51 kB 00:00 2024-04-22T01:53:21.570 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-22T01:53:21.587 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-toml-0.10.2-6.el9.noarch.rpm 55 kB/s | 42 kB 00:00 2024-04-22T01:53:21.603 INFO:teuthology.orchestra.run.smithi150.stdout:Package ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:53:21.637 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-04-22T01:53:21.648 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:21.649 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:53:21.649 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:21.654 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-04-22T01:53:21.687 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 588 kB/s | 19 kB 00:00 2024-04-22T01:53:21.714 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-04-22T01:53:21.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T01:53:21.738 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-04-22T01:53:21.796 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 556 kB/s | 46 kB 00:00 2024-04-22T01:53:21.822 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-04-22T01:53:21.864 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 396 kB/s | 16 kB 00:00 2024-04-22T01:53:21.922 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-04-22T01:53:21.948 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-22T01:53:21.973 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-04-22T01:53:22.006 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-cherrypy-18.6.1-2.el9.noarch.r 776 kB/s | 358 kB 00:00 2024-04-22T01:53:22.031 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-22T01:53:22.057 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-04-22T01:53:22.090 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-22T01:53:22.093 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:22.093 INFO:teuthology.orchestra.run.smithi107.stdout:Total 12 MB/s | 84 MB 00:07 2024-04-22T01:53:22.220 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:39 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:22.224 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:53:22.265 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:53:22.265 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:53:22.340 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 3.5 M 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:22.341 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 34 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 16 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:22.342 INFO:teuthology.orchestra.run.smithi150.stdout:Install 11 Packages 2024-04-22T01:53:22.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:22.343 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 7.6 M 2024-04-22T01:53:22.343 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 100 M 2024-04-22T01:53:22.343 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:22.872 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:53:22.873 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:53:23.242 INFO:teuthology.orchestra.run.smithi150.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3159.g676 85 kB/s | 16 kB 00:00 2024-04-22T01:53:23.342 INFO:teuthology.orchestra.run.smithi150.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3159.g67 116 kB/s | 34 kB 00:00 2024-04-22T01:53:23.376 INFO:teuthology.orchestra.run.smithi150.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-04-22T01:53:23.501 INFO:teuthology.orchestra.run.smithi150.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-04-22T01:53:23.710 INFO:teuthology.orchestra.run.smithi150.stdout:(5/11): ceph-mgr-dashboard-19.0.0-3159.g676947e 5.4 MB/s | 3.5 MB 00:00 2024-04-22T01:53:23.835 INFO:teuthology.orchestra.run.smithi150.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 58 kB/s | 19 kB 00:00 2024-04-22T01:53:24.118 INFO:teuthology.orchestra.run.smithi150.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 762 kB/s | 548 kB 00:00 2024-04-22T01:53:24.144 INFO:teuthology.orchestra.run.smithi150.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 468 kB/s | 144 kB 00:00 2024-04-22T01:53:24.211 INFO:teuthology.orchestra.run.smithi150.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 334 kB/s | 31 kB 00:00 2024-04-22T01:53:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-04-22T01:53:24.425 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:53:24.493 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T01:53:24.528 INFO:teuthology.orchestra.run.smithi150.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-04-22T01:53:24.530 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:24.530 INFO:teuthology.orchestra.run.smithi150.stdout:Total 3.5 MB/s | 7.6 MB 00:02 2024-04-22T01:53:24.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T01:53:24.561 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:24.562 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T01:53:24.577 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:24.577 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:24.722 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:24.723 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:24.750 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T01:53:24.794 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T01:53:24.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T01:53:24.920 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T01:53:25.005 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T01:53:25.047 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T01:53:25.081 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T01:53:25.111 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T01:53:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T01:53:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T01:53:25.198 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:25.219 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T01:53:25.254 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T01:53:25.276 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T01:53:25.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T01:53:25.297 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T01:53:25.326 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T01:53:25.375 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T01:53:25.401 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T01:53:25.510 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T01:53:25.547 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T01:53:25.559 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T01:53:25.645 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T01:53:25.657 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T01:53:25.695 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T01:53:25.712 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T01:53:25.760 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T01:53:25.764 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:53:25.798 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 9/11 2024-04-22T01:53:25.809 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T01:53:25.873 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T01:53:26.059 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T01:53:26.114 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T01:53:26.185 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T01:53:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T01:53:26.313 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T01:53:26.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T01:53:26.622 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 31/36 2024-04-22T01:53:26.649 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:53:26.673 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:26.852 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 10/11 2024-04-22T01:53:26.889 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 1/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 2/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 3/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T01:53:28.086 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T01:53:28.356 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:53:28.357 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:53:28.358 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:28.358 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:28.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T01:53:29.129 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:46 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:29.253 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:29.254 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-22T01:53:29.254 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:29.254 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-22T01:53:29.254 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 7.4 M 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T01:53:29.255 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout:Install 12 Packages 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 38 M 2024-04-22T01:53:29.256 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 209 M 2024-04-22T01:53:29.257 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:29.961 INFO:teuthology.orchestra.run.smithi150.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-04-22T01:53:30.187 INFO:teuthology.orchestra.run.smithi150.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 33 kB 00:00 2024-04-22T01:53:30.211 INFO:teuthology.orchestra.run.smithi150.stdout:(3/12): ceph-mgr-diskprediction-local-19.0.0-31 9.7 MB/s | 7.4 MB 00:00 2024-04-22T01:53:30.270 INFO:teuthology.orchestra.run.smithi150.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 982 kB/s | 803 kB 00:00 2024-04-22T01:53:30.353 INFO:teuthology.orchestra.run.smithi150.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-04-22T01:53:30.386 INFO:teuthology.orchestra.run.smithi150.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-04-22T01:53:30.512 INFO:teuthology.orchestra.run.smithi150.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-04-22T01:53:30.678 INFO:teuthology.orchestra.run.smithi150.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-04-22T01:53:30.745 INFO:teuthology.orchestra.run.smithi150.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-04-22T01:53:31.121 INFO:teuthology.orchestra.run.smithi150.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 6.2 MB/s | 4.7 MB 00:00 2024-04-22T01:53:31.321 INFO:teuthology.orchestra.run.smithi150.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.3 MB/s | 5.1 MB 00:00 2024-04-22T01:53:31.912 INFO:teuthology.orchestra.run.smithi150.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-04-22T01:53:31.913 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:31.914 INFO:teuthology.orchestra.run.smithi150.stdout:Total 14 MB/s | 38 MB 00:02 2024-04-22T01:53:32.262 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:32.289 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:32.289 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:32.698 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:32.698 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:33.506 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:33.584 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T01:53:33.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:53:33.736 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:53:33.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T01:53:33.957 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T01:53:33.986 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T01:53:34.063 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T01:53:34.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T01:53:34.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T01:53:36.047 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:53:37.097 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T01:53:37.127 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:53:37.524 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:53:37.550 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:53:37.550 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:37.550 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T01:53:37.551 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:53:37.551 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:53:37.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:38.091 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:53:38.114 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:38.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:53:38.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:38.453 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:53:39.646 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 1/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:53:39.647 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:53:40.072 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:40.073 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:40.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-rook 2024-04-22T01:53:41.002 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:01:58 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:41.123 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:41.124 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:41.124 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:41.124 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 49 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T01:53:41.125 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout:Install 13 Packages 2024-04-22T01:53:41.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:41.127 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 2.2 M 2024-04-22T01:53:41.127 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 26 M 2024-04-22T01:53:41.127 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:41.361 INFO:teuthology.orchestra.run.smithi150.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 200 kB/s | 19 kB 00:00 2024-04-22T01:53:41.386 INFO:teuthology.orchestra.run.smithi150.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 212 kB/s | 26 kB 00:00 2024-04-22T01:53:41.470 INFO:teuthology.orchestra.run.smithi150.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-04-22T01:53:41.495 INFO:teuthology.orchestra.run.smithi150.stdout:(4/13): ceph-mgr-rook-19.0.0-3159.g676947e1.el9 214 kB/s | 49 kB 00:00 2024-04-22T01:53:41.520 INFO:teuthology.orchestra.run.smithi150.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-22T01:53:41.545 INFO:teuthology.orchestra.run.smithi150.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-04-22T01:53:41.613 INFO:teuthology.orchestra.run.smithi150.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 458 kB/s | 54 kB 00:00 2024-04-22T01:53:41.788 INFO:teuthology.orchestra.run.smithi150.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-04-22T01:53:41.821 INFO:teuthology.orchestra.run.smithi150.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-04-22T01:53:42.021 INFO:teuthology.orchestra.run.smithi150.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 296 kB/s | 59 kB 00:00 2024-04-22T01:53:42.088 INFO:teuthology.orchestra.run.smithi150.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 426 kB/s | 203 kB 00:00 2024-04-22T01:53:42.189 INFO:teuthology.orchestra.run.smithi150.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 536 kB/s | 90 kB 00:00 2024-04-22T01:53:42.314 INFO:teuthology.orchestra.run.smithi150.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-04-22T01:53:42.315 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:42.315 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-04-22T01:53:42.385 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:42.401 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:42.401 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:42.487 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:42.487 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:42.984 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:43.154 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T01:53:43.207 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T01:53:43.270 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T01:53:43.308 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T01:53:43.349 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T01:53:43.443 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T01:53:43.552 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T01:53:43.601 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T01:53:44.320 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T01:53:44.365 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T01:53:44.398 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T01:53:44.446 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T01:53:44.468 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:53:45.108 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:53:45.108 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 1/36 2024-04-22T01:53:45.108 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2/36 2024-04-22T01:53:45.108 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 3/36 2024-04-22T01:53:45.108 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 4/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 5/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 6/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T01:53:45.109 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T01:53:45.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T01:53:45.111 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T01:53:45.111 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T01:53:45.111 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 1/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T01:53:45.162 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T01:53:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T01:53:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T01:53:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T01:53:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T01:53:45.163 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T01:53:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T01:53:45.463 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:45.463 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:53:45.464 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:45.465 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:45.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T01:53:46.176 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T01:53:46.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:53:46.177 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:53:46.178 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:53:46.179 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:53:46.180 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:53:46.180 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:46.180 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:46.233 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:03 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:46.354 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:46.354 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:46.354 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 142 k 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:46.355 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:46.356 INFO:teuthology.orchestra.run.smithi150.stdout:Install 6 Packages 2024-04-22T01:53:46.356 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:46.356 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 7.0 M 2024-04-22T01:53:46.356 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 33 M 2024-04-22T01:53:46.357 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:46.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-04-22T01:53:46.891 INFO:teuthology.orchestra.run.smithi150.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3159.g676947e1.e 456 kB/s | 142 kB 00:00 2024-04-22T01:53:46.996 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:34 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:47.073 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:53:47.083 INFO:teuthology.orchestra.run.smithi150.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 492 kB/s | 248 kB 00:00 2024-04-22T01:53:47.115 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:47.116 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:53:47.116 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:47.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-04-22T01:53:47.367 INFO:teuthology.orchestra.run.smithi150.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-04-22T01:53:47.392 INFO:teuthology.orchestra.run.smithi150.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-04-22T01:53:47.442 INFO:teuthology.orchestra.run.smithi150.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 678 kB/s | 51 kB 00:00 2024-04-22T01:53:47.686 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:47.805 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 762 k 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:47.806 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 762 k 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 769 k 2024-04-22T01:53:47.807 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:53:48.092 INFO:teuthology.orchestra.run.smithi150.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-04-22T01:53:48.094 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:48.095 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-04-22T01:53:48.107 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-19.0.0-3159.g676947e1.el9.noarch.rpm 2.5 MB/s | 762 kB 00:00 2024-04-22T01:53:48.109 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:48.109 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.5 MB/s | 762 kB 00:00 2024-04-22T01:53:48.109 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:53:48.115 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:53:48.115 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:53:48.121 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:53:48.121 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:53:48.167 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:48.178 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:48.178 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:48.197 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:53:48.274 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:48.274 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:48.580 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:48.653 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:48.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:48.754 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T01:53:48.803 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T01:53:49.070 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:49.312 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:53:49.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:49.312 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:53:49.313 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:49.313 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:49.313 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:49.329 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T01:53:49.423 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T01:53:49.496 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T01:53:49.525 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:53:49.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T01:53:50.070 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:50.191 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:50.192 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:50.192 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-04-22T01:53:50.192 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3159.g676947e1.el9 ceph 157 k 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 157 k 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 640 k 2024-04-22T01:53:50.193 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:53:50.262 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:53:50.262 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/6 2024-04-22T01:53:50.263 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T01:53:50.263 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T01:53:50.263 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T01:53:50.263 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:50.505 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:50.687 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-19.0.0-3159.g676947 318 kB/s | 157 kB 00:00 2024-04-22T01:53:50.688 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:50.689 INFO:teuthology.orchestra.run.smithi107.stdout:Total 317 kB/s | 157 kB 00:00 2024-04-22T01:53:50.689 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:53:50.698 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:53:50.698 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:53:50.755 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:53:50.755 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:53:50.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-fuse 2024-04-22T01:53:50.908 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:53:50.997 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:51.021 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:51.021 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:51.021 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:53:51.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:51.285 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:08 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:51.406 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 871 k 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T01:53:51.407 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 951 k 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 4.0 M 2024-04-22T01:53:51.408 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:51.671 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 585 kB/s | 80 kB 00:00 2024-04-22T01:53:51.841 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:53:51.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:51.842 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:53:51.842 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:51.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:51.842 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:52.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-04-22T01:53:52.321 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-fuse-19.0.0-3159.g676947e1.el9.x86_ 1.1 MB/s | 871 kB 00:00 2024-04-22T01:53:52.322 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:52.322 INFO:teuthology.orchestra.run.smithi150.stdout:Total 1.0 MB/s | 951 kB 00:00 2024-04-22T01:53:52.334 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:52.359 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:52.359 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:52.440 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:52.441 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:52.545 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:52.623 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:53:52.626 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:52.666 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:52.667 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:53:52.667 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:52.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T01:53:52.774 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T01:53:52.836 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:53:53.267 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:53.386 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:53.387 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:53.387 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:53:53.387 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:53.387 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 3.5 M 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 34 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 16 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T01:53:53.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout:Install 11 Packages 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:53.389 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 7.6 M 2024-04-22T01:53:53.390 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 100 M 2024-04-22T01:53:53.390 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:53:53.489 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:53:53.489 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/2 2024-04-22T01:53:53.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T01:53:53.746 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:53.746 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:53.746 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T01:53:53.746 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:53.747 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:54.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-volume 2024-04-22T01:53:54.074 INFO:teuthology.orchestra.run.smithi107.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3159.g67 134 kB/s | 34 kB 00:00 2024-04-22T01:53:54.100 INFO:teuthology.orchestra.run.smithi107.stdout:(2/11): ceph-prometheus-alerts-19.0.0-3159.g676 59 kB/s | 16 kB 00:00 2024-04-22T01:53:54.350 INFO:teuthology.orchestra.run.smithi107.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-04-22T01:53:54.375 INFO:teuthology.orchestra.run.smithi107.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 896 kB/s | 269 kB 00:00 2024-04-22T01:53:54.442 INFO:teuthology.orchestra.run.smithi107.stdout:(5/11): ceph-mgr-dashboard-19.0.0-3159.g676947e 5.7 MB/s | 3.5 MB 00:00 2024-04-22T01:53:54.569 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:11 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:54.690 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:54.694 INFO:teuthology.orchestra.run.smithi107.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 61 kB/s | 19 kB 00:00 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 273 k 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout:Install 2 Packages 2024-04-22T01:53:54.695 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:54.696 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 350 k 2024-04-22T01:53:54.696 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 1.5 M 2024-04-22T01:53:54.696 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:54.859 INFO:teuthology.orchestra.run.smithi107.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-04-22T01:53:54.885 INFO:teuthology.orchestra.run.smithi107.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 753 kB/s | 144 kB 00:00 2024-04-22T01:53:54.927 INFO:teuthology.orchestra.run.smithi150.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 679 kB/s | 77 kB 00:00 2024-04-22T01:53:54.943 INFO:teuthology.orchestra.run.smithi107.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 368 kB/s | 31 kB 00:00 2024-04-22T01:53:55.010 INFO:teuthology.orchestra.run.smithi107.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-04-22T01:53:55.068 INFO:teuthology.orchestra.run.smithi150.stdout:(2/2): ceph-volume-19.0.0-3159.g676947e1.el9.no 1.0 MB/s | 273 kB 00:00 2024-04-22T01:53:55.069 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:55.069 INFO:teuthology.orchestra.run.smithi150.stdout:Total 928 kB/s | 350 kB 00:00 2024-04-22T01:53:55.080 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:55.085 INFO:teuthology.orchestra.run.smithi107.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-04-22T01:53:55.090 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:55.090 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.5 MB/s | 7.6 MB 00:01 2024-04-22T01:53:55.091 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:55.091 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:55.117 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:55.117 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:55.119 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:53:55.136 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:53:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:53:55.260 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:55.282 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:53:55.284 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:53:55.451 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T01:53:55.488 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:53:55.503 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:53:55.503 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:53:55.503 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T01:53:55.503 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:55.766 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:53:55.842 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T01:53:55.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T01:53:55.957 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T01:53:56.091 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T01:53:56.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T01:53:56.204 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 1/2 2024-04-22T01:53:56.226 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T01:53:56.269 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T01:53:56.297 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:53:56.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 9/11 2024-04-22T01:53:56.479 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:56.480 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:56.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados-devel 2024-04-22T01:53:57.298 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:14 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:57.379 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 10/11 2024-04-22T01:53:57.421 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 125 k 2024-04-22T01:53:57.423 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 125 k 2024-04-22T01:53:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 445 k 2024-04-22T01:53:57.425 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:53:57.741 INFO:teuthology.orchestra.run.smithi150.stdout:librados-devel-19.0.0-3159.g676947e1.el9.x86_64 395 kB/s | 125 kB 00:00 2024-04-22T01:53:57.741 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:53:57.742 INFO:teuthology.orchestra.run.smithi150.stdout:Total 393 kB/s | 125 kB 00:00 2024-04-22T01:53:57.742 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:53:57.747 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:53:57.748 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:53:57.811 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:53:57.811 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:53:57.960 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:53:58.062 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:53:58.497 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:53:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:53:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 1/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 2/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 3/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T01:53:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:53:58.731 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:58.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs2 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:59.020 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:59.021 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:53:59.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T01:53:59.494 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:53:59.575 INFO:teuthology.orchestra.run.smithi150.stdout:Package libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:53:59.616 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:53:59.617 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:53:59.617 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:53:59.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install libcephfs-devel 2024-04-22T01:53:59.769 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:47 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:53:59.890 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:53:59.891 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-22T01:53:59.891 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:53:59.891 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-22T01:53:59.891 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 7.4 M 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T01:53:59.892 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 38 M 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 209 M 2024-04-22T01:53:59.893 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:00.194 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:17 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:00.313 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 32 k 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:00.314 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:54:00.315 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:00.315 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 32 k 2024-04-22T01:54:00.315 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 142 k 2024-04-22T01:54:00.315 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:54:00.451 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 775 kB/s | 195 kB 00:00 2024-04-22T01:54:00.478 INFO:teuthology.orchestra.run.smithi150.stdout:libcephfs-devel-19.0.0-3159.g676947e1.el9.x86_6 195 kB/s | 32 kB 00:00 2024-04-22T01:54:00.478 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:00.479 INFO:teuthology.orchestra.run.smithi150.stdout:Total 193 kB/s | 32 kB 00:00 2024-04-22T01:54:00.479 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:54:00.482 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:54:00.482 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:54:00.485 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:54:00.485 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:54:00.501 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 655 kB/s | 33 kB 00:00 2024-04-22T01:54:00.565 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:54:00.593 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-22T01:54:00.652 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 313 kB/s | 18 kB 00:00 2024-04-22T01:54:00.665 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:00.702 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-22T01:54:00.811 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-04-22T01:54:00.894 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-22T01:54:01.014 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:01.070 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-04-22T01:54:01.145 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:01.202 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:01.270 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-22T01:54:01.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librados2 2024-04-22T01:54:01.745 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-04-22T01:54:01.873 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:18 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:01.952 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:01.993 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:01.994 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:01.994 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:02.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install librbd1 2024-04-22T01:54:02.568 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:02.647 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:02.688 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:02.689 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:02.689 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:02.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rados 2024-04-22T01:54:03.263 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:20 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:03.343 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:03.385 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:03.386 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:03.386 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:03.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rgw 2024-04-22T01:54:03.704 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-3 2.1 MB/s | 7.4 MB 00:03 2024-04-22T01:54:03.705 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:03.705 INFO:teuthology.orchestra.run.smithi107.stdout:Total 10 MB/s | 38 MB 00:03 2024-04-22T01:54:03.959 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:20 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:04.038 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:04.062 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:04.079 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:04.080 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:04.080 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:04.088 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:04.088 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:04.140 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-cephfs 2024-04-22T01:54:04.500 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:04.500 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:04.653 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:04.733 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:04.775 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:04.776 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:04.776 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:04.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install python3-rbd 2024-04-22T01:54:05.317 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:05.358 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:05.375 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T01:54:05.410 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:54:05.438 INFO:teuthology.orchestra.run.smithi150.stdout:Package python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:05.479 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:05.480 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-22T01:54:05.480 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:05.511 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:54:05.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-fuse 2024-04-22T01:54:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T01:54:05.739 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T01:54:05.768 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T01:54:05.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T01:54:06.046 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:23 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:06.165 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-22T01:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:54:06.166 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 90 k 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 90 k 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 313 k 2024-04-22T01:54:06.167 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:54:06.490 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T01:54:06.564 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-fuse-19.0.0-3159.g676947e1.el9.x86_64.rpm 226 kB/s | 90 kB 00:00 2024-04-22T01:54:06.565 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:06.565 INFO:teuthology.orchestra.run.smithi150.stdout:Total 225 kB/s | 90 kB 00:00 2024-04-22T01:54:06.565 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:54:06.574 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:54:06.574 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:54:06.615 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T01:54:06.637 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:54:06.637 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:54:06.787 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:54:06.876 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:07.239 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:07.498 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:07.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-mirror 2024-04-22T01:54:07.844 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:54:08.315 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:08.440 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repo Size 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 3.4 M 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 29 M 2024-04-22T01:54:08.441 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:54:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T01:54:08.909 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:54:09.244 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-mirror-19.0.0-3159.g676947e1.el9.x86_64.rpm 4.2 MB/s | 3.4 MB 00:00 2024-04-22T01:54:09.245 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:09.245 INFO:teuthology.orchestra.run.smithi150.stdout:Total 4.2 MB/s | 3.4 MB 00:00 2024-04-22T01:54:09.245 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:54:09.256 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:54:09.256 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:54:09.338 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:54:09.339 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:54:09.909 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:54:09.970 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:09.994 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:09.995 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:54:09.995 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:54:09.995 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:54:09.995 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:54:09.995 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:11.182 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 1/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T01:54:11.432 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T01:54:11.433 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T01:54:11.433 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:54:11.433 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T01:54:11.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install rbd-nbd 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:54:11.851 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:11.852 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:11.961 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:02:28 ago on Mon 22 Apr 2024 01:51:43 AM UTC. 2024-04-22T01:54:12.085 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 175 k 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:12.086 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout:Install 1 Package 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 175 k 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 673 k 2024-04-22T01:54:12.087 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-22T01:54:12.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-04-22T01:54:12.314 INFO:teuthology.orchestra.run.smithi150.stdout:rbd-nbd-19.0.0-3159.g676947e1.el9.x86_64.rpm 774 kB/s | 175 kB 00:00 2024-04-22T01:54:12.315 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:12.315 INFO:teuthology.orchestra.run.smithi150.stdout:Total 769 kB/s | 175 kB 00:00 2024-04-22T01:54:12.315 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-22T01:54:12.324 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-22T01:54:12.324 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-22T01:54:12.388 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-22T01:54:12.389 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-22T01:54:12.540 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-22T01:54:12.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:12.779 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:00 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:12.901 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 49 k 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T01:54:12.902 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-04-22T01:54:12.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:12.904 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.2 M 2024-04-22T01:54:12.904 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-04-22T01:54:12.904 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:13.010 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:13.143 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 275 kB/s | 26 kB 00:00 2024-04-22T01:54:13.160 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 174 kB/s | 19 kB 00:00 2024-04-22T01:54:13.219 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-04-22T01:54:13.244 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-04-22T01:54:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:13.251 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-22T01:54:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:13.251 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:13.252 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-22T01:54:13.269 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): ceph-mgr-rook-19.0.0-3159.g676947e1.el9 225 kB/s | 49 kB 00:00 2024-04-22T01:54:13.295 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-22T01:54:13.328 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-04-22T01:54:13.475 DEBUG:teuthology.parallel:result is None 2024-04-22T01:54:13.495 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 142 kB/s | 32 kB 00:00 2024-04-22T01:54:13.521 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-04-22T01:54:13.629 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-22T01:54:13.679 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 668 kB/s | 203 kB 00:00 2024-04-22T01:54:13.713 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-04-22T01:54:13.746 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-04-22T01:54:13.749 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:13.749 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-04-22T01:54:13.811 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:13.825 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:13.825 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:13.907 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:13.907 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:14.414 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:14.585 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T01:54:14.647 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T01:54:14.711 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T01:54:14.749 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T01:54:14.790 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T01:54:14.883 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T01:54:14.985 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T01:54:15.033 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T01:54:15.752 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T01:54:15.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T01:54:15.840 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T01:54:15.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T01:54:15.909 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 1/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T01:54:16.672 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T01:54:16.673 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T01:54:16.988 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:54:16.989 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:54:16.990 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:54:16.990 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:54:16.990 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:54:16.990 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:16.990 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:17.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T01:54:17.746 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:05 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:17.866 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:17.866 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 142 k 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T01:54:17.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-04-22T01:54:17.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:17.869 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 7.0 M 2024-04-22T01:54:17.869 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-04-22T01:54:17.869 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:18.474 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-04-22T01:54:18.591 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3159.g676947e1.e 470 kB/s | 142 kB 00:00 2024-04-22T01:54:18.683 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.6 MB/s | 548 kB 00:00 2024-04-22T01:54:18.716 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-04-22T01:54:18.742 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-04-22T01:54:19.675 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-04-22T01:54:19.676 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:19.676 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-04-22T01:54:19.745 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:19.756 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:19.757 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:19.852 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:19.854 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:20.151 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:20.324 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T01:54:20.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T01:54:20.908 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T01:54:20.988 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T01:54:21.059 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T01:54:21.089 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:54:21.882 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:54:21.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/6 2024-04-22T01:54:21.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T01:54:21.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T01:54:21.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T01:54:21.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T01:54:22.145 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T01:54:22.145 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:22.146 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:22.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-04-22T01:54:22.881 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:10 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:23.003 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:23.003 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 871 k 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-04-22T01:54:23.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:23.005 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 951 k 2024-04-22T01:54:23.005 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 4.0 M 2024-04-22T01:54:23.005 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:23.318 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 425 kB/s | 80 kB 00:00 2024-04-22T01:54:23.411 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-19.0.0-3159.g676947e1.el9.x86_ 3.0 MB/s | 871 kB 00:00 2024-04-22T01:54:23.411 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:23.411 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.3 MB/s | 951 kB 00:00 2024-04-22T01:54:23.422 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:23.523 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:23.524 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:23.701 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:23.864 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T01:54:23.935 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:54:24.593 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:54:24.593 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/2 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:24.860 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:25.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2024-04-22T01:54:25.642 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:13 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:25.764 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 273 k 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-04-22T01:54:25.765 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:25.766 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 350 k 2024-04-22T01:54:25.767 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.5 M 2024-04-22T01:54:25.767 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:26.149 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 685 kB/s | 77 kB 00:00 2024-04-22T01:54:26.424 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-volume-19.0.0-3159.g676947e1.el9.no 704 kB/s | 273 kB 00:00 2024-04-22T01:54:26.425 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:26.425 INFO:teuthology.orchestra.run.smithi107.stdout:Total 531 kB/s | 350 kB 00:00 2024-04-22T01:54:26.437 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:26.447 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:26.447 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:26.473 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:26.474 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:26.632 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T01:54:26.852 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:54:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:54:26.867 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:54:26.867 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T01:54:26.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 1/2 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:27.922 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:28.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-04-22T01:54:28.716 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:28.838 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 125 k 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:28.839 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 125 k 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 445 k 2024-04-22T01:54:28.840 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:29.155 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-19.0.0-3159.g676947e1.el9.x86_64 397 kB/s | 125 kB 00:00 2024-04-22T01:54:29.156 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:29.156 INFO:teuthology.orchestra.run.smithi107.stdout:Total 394 kB/s | 125 kB 00:00 2024-04-22T01:54:29.156 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:29.162 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:29.162 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:29.226 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:29.226 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:29.378 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:29.468 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:29.878 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:30.102 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:30.103 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:30.103 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:30.103 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:30.103 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:30.103 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:30.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-04-22T01:54:30.880 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:18 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:30.959 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:31.001 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:31.002 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:31.002 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:31.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-04-22T01:54:31.600 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:31.718 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:31.719 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:31.719 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-04-22T01:54:31.719 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 32 k 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 32 k 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 142 k 2024-04-22T01:54:31.720 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:31.966 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-19.0.0-3159.g676947e1.el9.x86_6 129 kB/s | 32 kB 00:00 2024-04-22T01:54:31.967 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:31.967 INFO:teuthology.orchestra.run.smithi107.stdout:Total 128 kB/s | 32 kB 00:00 2024-04-22T01:54:31.967 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:31.970 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:31.970 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:31.972 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:31.973 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:32.059 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:32.170 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:32.496 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:32.697 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:32.698 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:32.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:32.698 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:32.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-04-22T01:54:33.377 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:33.456 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:33.498 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:33.499 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:33.499 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:33.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-04-22T01:54:34.077 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:34.155 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:34.196 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:34.196 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:34.197 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:34.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-04-22T01:54:34.775 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:34.853 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:34.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-04-22T01:54:35.487 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:23 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:35.566 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:35.607 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:35.608 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:35.608 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:35.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-04-22T01:54:36.194 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:24 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:36.272 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:36.313 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:36.314 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:36.314 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:36.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-04-22T01:54:36.901 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:24 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:36.979 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:54:37.020 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:37.021 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-04-22T01:54:37.021 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:37.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-04-22T01:54:37.598 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:37.717 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:37.718 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:37.718 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-22T01:54:37.718 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 90 k 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:37.719 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 90 k 2024-04-22T01:54:37.720 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 313 k 2024-04-22T01:54:37.720 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:38.226 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-19.0.0-3159.g676947e1.el9.x86_64.rpm 177 kB/s | 90 kB 00:00 2024-04-22T01:54:38.227 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:38.228 INFO:teuthology.orchestra.run.smithi107.stdout:Total 176 kB/s | 90 kB 00:00 2024-04-22T01:54:38.228 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:38.236 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:38.236 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:38.299 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:38.299 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:38.440 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:38.558 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:38.890 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:39.117 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:39.118 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:39.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-04-22T01:54:39.921 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:27 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:40.043 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:40.044 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.4 M 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 29 M 2024-04-22T01:54:40.045 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:40.454 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-19.0.0-3159.g676947e1.el9.x86_64.rpm 8.3 MB/s | 3.4 MB 00:00 2024-04-22T01:54:40.455 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:40.456 INFO:teuthology.orchestra.run.smithi107.stdout:Total 8.3 MB/s | 3.4 MB 00:00 2024-04-22T01:54:40.456 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:40.466 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:40.466 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:40.547 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:40.548 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:41.172 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:41.196 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:41.196 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:54:41.196 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:54:41.197 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:54:41.197 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:54:41.197 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:42.365 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:42.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-04-22T01:54:43.097 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:31 ago on Mon 22 Apr 2024 01:52:12 AM UTC. 2024-04-22T01:54:43.220 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 175 k 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:43.221 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 175 k 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 673 k 2024-04-22T01:54:43.222 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-04-22T01:54:43.598 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-19.0.0-3159.g676947e1.el9.x86_64.rpm 466 kB/s | 175 kB 00:00 2024-04-22T01:54:43.599 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:43.599 INFO:teuthology.orchestra.run.smithi107.stdout:Total 464 kB/s | 175 kB 00:00 2024-04-22T01:54:43.599 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-04-22T01:54:43.609 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-04-22T01:54:43.609 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-04-22T01:54:43.672 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-04-22T01:54:43.673 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-04-22T01:54:43.822 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-04-22T01:54:43.944 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:44.340 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:44.607 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:54:44.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:44.608 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-04-22T01:54:44.608 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:44.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:44.608 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-04-22T01:54:44.856 DEBUG:teuthology.parallel:result is None 2024-04-22T01:54:44.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:54:45.115 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:54:45.138 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-3159.g676947e1.el9 2024-04-22T01:54:45.140 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3159.g676947e1.el9 2024-04-22T01:54:45.141 INFO:teuthology.task.install:The correct ceph version 19.0.0-3159.g676947e1 is installed. 2024-04-22T01:54:45.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:54:45.365 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:54:45.390 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-3159.g676947e1.el9 2024-04-22T01:54:45.391 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3159.g676947e1.el9 2024-04-22T01:54:45.391 INFO:teuthology.task.install:The correct ceph version 19.0.0-3159.g676947e1 is installed. 2024-04-22T01:54:45.392 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T01:54:45.393 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:45.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:54:45.416 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:45.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:54:45.459 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T01:54:45.459 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:45.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:54:45.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:54:45.546 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:45.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:54:45.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:54:45.637 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T01:54:45.638 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:45.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:54:45.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:54:45.737 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:45.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:54:45.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:54:45.828 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T01:54:45.829 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:45.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:54:45.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:54:45.915 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:45.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:54:45.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:54:46.002 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T01:54:46.087 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T01:54:46.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2024-04-22T01:54:46.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-04-22T01:54:46.113 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T01:54:46.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T01:54:46.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T01:54:46.177 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T01:54:46.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T01:54:46.217 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T01:54:46.251 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T01:54:46.251 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T01:54:46.251 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7668104', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKktq2SK+j+hRsjA8azgEg0Ch7sfdBKBzBmFRSDqNg+A+9nIkMS2Pj/FeT2gIg7W6d0c/VDs8uku7iiOfbovN4=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6fG8tz02/G/GPRaj1/U74lFo8yZzXxkcBYrMbQh8/NNLzWU098PBo/Lux++g9b6vl3k1FG/SmkSl2fQFdkDrY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'concurrency': 128, 'size': 8192, 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_01:37:40', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T01:54:46.252 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T01:54:46.280 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T01:54:46.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T01:54:46.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T01:54:46.360 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:46.360 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:54:46.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:54:46.414 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:46.467 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-22 01:54:41.456054206 +0000 2024-04-22T01:54:46.468 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-22 01:51:04.896507264 +0000 2024-04-22T01:54:46.468 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-22 01:51:04.896507264 +0000 2024-04-22T01:54:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-22 01:51:04.896507264 +0000 2024-04-22T01:54:46.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:54:46.530 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-22T01:54:46.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-22T01:54:46.531 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00018539 s, 2.8 MB/s 2024-04-22T01:54:46.531 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:54:46.586 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-22 01:54:41.456054206 +0000 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-22 01:51:05.230501409 +0000 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-22 01:51:05.230501409 +0000 2024-04-22T01:54:46.642 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-22 01:51:05.230501409 +0000 2024-04-22T01:54:46.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:54:46.703 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-22T01:54:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-22T01:54:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000207291 s, 2.5 MB/s 2024-04-22T01:54:46.705 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:54:46.760 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-04-22T01:54:46.815 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-22 01:54:41.456054206 +0000 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-22 01:51:05.564495555 +0000 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-22 01:51:05.564495555 +0000 2024-04-22T01:54:46.816 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-22 01:51:05.564495555 +0000 2024-04-22T01:54:46.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:54:46.878 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-22T01:54:46.878 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-22T01:54:46.878 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000162855 s, 3.1 MB/s 2024-04-22T01:54:46.879 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:54:46.934 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-04-22 01:54:41.457054188 +0000 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-04-22 01:51:05.893489788 +0000 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-04-22 01:51:05.893489788 +0000 2024-04-22T01:54:46.989 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-04-22 01:51:05.893489788 +0000 2024-04-22T01:54:46.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:54:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-04-22T01:54:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-04-22T01:54:47.050 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000235422 s, 2.2 MB/s 2024-04-22T01:54:47.051 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:54:47.107 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T01:54:47.108 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:47.108 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:54:47.123 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:54:47.123 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-04-22T01:54:47.178 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:54:47.178 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:47.178 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-22 01:54:10.235261938 +0000 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-22 01:51:04.891839571 +0000 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-22 01:51:04.891839571 +0000 2024-04-22T01:54:47.179 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-22 01:51:04.891839571 +0000 2024-04-22T01:54:47.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:54:47.241 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-22T01:54:47.241 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-22T01:54:47.241 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000204118 s, 2.5 MB/s 2024-04-22T01:54:47.242 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:54:47.297 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-22 01:54:10.235261938 +0000 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-22 01:51:05.234832475 +0000 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-22 01:51:05.234832475 +0000 2024-04-22T01:54:47.352 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-22 01:51:05.234832475 +0000 2024-04-22T01:54:47.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:54:47.414 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-22T01:54:47.414 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-22T01:54:47.415 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00020008 s, 2.6 MB/s 2024-04-22T01:54:47.416 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:54:47.471 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-04-22T01:54:47.526 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:54:47.526 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:47.526 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T01:54:47.526 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:47.527 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:47.527 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-22 01:54:10.235261938 +0000 2024-04-22T01:54:47.527 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-22 01:51:05.580825319 +0000 2024-04-22T01:54:47.527 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-22 01:51:05.580825319 +0000 2024-04-22T01:54:47.527 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-22 01:51:05.580825319 +0000 2024-04-22T01:54:47.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:54:47.588 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-22T01:54:47.588 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-22T01:54:47.588 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000193968 s, 2.6 MB/s 2024-04-22T01:54:47.589 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:54:47.645 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-22 01:54:10.236261917 +0000 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-22 01:51:05.932818038 +0000 2024-04-22T01:54:47.700 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-22 01:51:05.932818038 +0000 2024-04-22T01:54:47.701 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-22 01:51:05.932818038 +0000 2024-04-22T01:54:47.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:54:47.762 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-22T01:54:47.762 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-22T01:54:47.762 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000170107 s, 3.0 MB/s 2024-04-22T01:54:47.763 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:54:47.820 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T01:54:47.820 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-22T01:54:47.821 INFO:tasks.ceph:Generating config... 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T01:54:47.822 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] log to file = False 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T01:54:47.823 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T01:54:47.824 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-04-22T01:54:47.824 INFO:tasks.ceph:[client.0] debug ms = 1 2024-04-22T01:54:47.824 INFO:tasks.ceph:[client.0] debug objecter = 20 2024-04-22T01:54:47.824 INFO:tasks.ceph:[client.0] debug rados = 20 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-22T01:54:47.824 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-22T01:54:47.825 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-22T01:54:47.825 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-04-22T01:54:47.825 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-22T01:54:47.825 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-22T01:54:47.825 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-22T01:54:47.826 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-22T01:54:47.826 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-22T01:54:47.826 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-22T01:54:47.826 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T01:54:47.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T01:54:47.886 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T01:54:47.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T01:54:47.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:54:47.962 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107')] 2024-04-22T01:54:47.962 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.107', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-22T01:54:47.963 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:47.963 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:54:48.017 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.107 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:48.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:54:48.106+0000 7fce0a666f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:54:48.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:54:48.106+0000 7fce0a666f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:54:48.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:54:48.106+0000 7fce0a666f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:54:48.107 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2024-04-22T01:54:48.108469+0000 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-04-22T01:54:48.108469+0000 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 17 (quincy) 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-04-22T01:54:48.108 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-22T01:54:48.110 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:54:48.124 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 62d4ad10-8dc5-4417-9af4-a9625064a333... 2024-04-22T01:54:48.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T01:54:48.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.205 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.206 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T01:54:48.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = 172.21.15.107 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: ms client mode = crc 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: ms cluster mode = crc 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon client mode = crc 2024-04-22T01:54:48.208 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon cluster mode = crc 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon service mode = crc 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: ms service mode = crc 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_min_size = 2 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_size = 3 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 120 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.209 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.210 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2024-04-22T01:54:48.213 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-04-22T01:54:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression algorithm = snappy 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression mode = aggressive 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore rocksdb cf = False 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: crimson osd obc lru size = 10 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: err to stderr = True 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: log flush on exit = True 2024-04-22T01:54:48.215 INFO:teuthology.orchestra.run.smithi107.stdout: log to file = False 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: log to stderr = True 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug reject backfill probability = 0.3 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd delete sleep = 1 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd max backfills = 3 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub min interval = 60 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: osd snap trim sleep = 2 2024-04-22T01:54:48.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2024-04-22T01:54:48.217 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2024-04-22T01:54:48.218 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2024-04-22T01:54:48.219 INFO:teuthology.orchestra.run.smithi107.stdout: mon min osdmap epochs = 50 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune interval = 2 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune min = 15 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: mon osdmap full prune txsize = 2 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default crimson = True 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: paxos service trim min = 10 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-04-22T01:54:48.220 INFO:teuthology.orchestra.run.smithi107.stdout: debug objecter = 20 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: debug rados = 20 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2024-04-22T01:54:48.221 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.223 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.224 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = 172.21.15.107 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: ms client mode = crc 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: ms cluster mode = crc 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: ms mon client mode = crc 2024-04-22T01:54:48.226 INFO:teuthology.orchestra.run.smithi150.stdout: ms mon cluster mode = crc 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: ms mon service mode = crc 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: ms service mode = crc 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_min_size = 2 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_size = 3 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 120 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-22T01:54:48.227 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-22T01:54:48.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore compression algorithm = snappy 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore compression mode = aggressive 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-22T01:54:48.229 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore rocksdb cf = False 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: crimson osd obc lru size = 10 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: err to stderr = True 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: log flush on exit = True 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: log to file = False 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: log to stderr = True 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T01:54:48.230 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd blocked scrub grace period = 3600 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug reject backfill probability = 0.3 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd delete sleep = 1 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd max backfills = 3 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd max markdown count = 1000 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub min interval = 60 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd shutdown pgref assert = True 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: osd snap trim sleep = 2 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.231 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-22T01:54:48.232 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-22T01:54:48.233 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: mon min osdmap epochs = 50 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune interval = 2 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune min = 15 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune txsize = 2 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default crimson = True 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: paxos service trim min = 10 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T01:54:48.234 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: debug objecter = 20 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: debug rados = 20 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-04-22T01:54:48.235 INFO:teuthology.orchestra.run.smithi150.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:54:48.235 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T01:54:48.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T01:54:48.326 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T01:54:48.326 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:48.326 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T01:54:48.340 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:48.341 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T01:54:48.396 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T01:54:48.396 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:48.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T01:54:48.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:54:48.468 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:54:48.468 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:48.523 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T01:54:48.523 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:48.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T01:54:48.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:54:48.557 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:54:48.557 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:48.612 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T01:54:48.612 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T01:54:48.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:54:48.716 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:54:48.718 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T01:54:48.719 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T01:54:48.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T01:54:48.766 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T01:54:48.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T01:54:48.827 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-22T01:54:48.839 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T01:54:48.839 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-04-22T01:54:48.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T01:54:48.862 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T01:54:48.863 INFO:tasks.ceph:role: osd.0 2024-04-22T01:54:48.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T01:54:48.863 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T01:54:48.933 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:54:48.934 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:54:49.442 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-04-22T01:54:49.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-04-22T01:54:49.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T01:54:49.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T01:54:49.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T01:54:49.669 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-04-22T01:54:49.669 INFO:tasks.ceph:role: osd.1 2024-04-22T01:54:49.669 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T01:54:49.669 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:54:49.740 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:54:50.205 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-04-22T01:54:50.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-04-22T01:54:50.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T01:54:50.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T01:54:50.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:50.619 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:54:50.620 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:50,621 seastar - Reactor backend: linux-aio 2024-04-22T01:54:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:==26598==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:54:50.866 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:50,867 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T01:54:50.866 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:50,868 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-22T01:54:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:DEBUG 2024-04-22 01:54:50,869 [shard 0:main] seastar - reactor::drain 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:51.457 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T01:54:51.512 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:54:51.515 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:51,517 seastar - Reactor backend: linux-aio 2024-04-22T01:54:51.525 INFO:teuthology.orchestra.run.smithi107.stderr:==26596==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:54:56.156 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:56.156 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-04-22T01:54:56.156 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-04-22T01:54:56.156 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-04-22T01:54:56.157 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:56.157 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T01:54:56.157 INFO:teuthology.orchestra.run.smithi107.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:54:56.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T01:54:56.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:54:56.487 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:54:56.488 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:56,489 seastar - Reactor backend: linux-aio 2024-04-22T01:54:56.492 INFO:teuthology.orchestra.run.smithi107.stderr:==26781==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:54:56.596 INFO:teuthology.orchestra.run.smithi107.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb98fc39 0xb862700 0xb863c7c 0xb863e92 0xb863fe8 0xb8644b1 0x3e6ef 0x2650f 0xef7ec 0x4a0a5 0x35b7c 0x3689e 0x5da2988 0x5da3481 0x5da38e6 0x5dff874 0x5d7e3f0 0x5d7e924 0x5bc3681 0x5bde745 0x6eb8f84 0x6eca757 0x6ecbbf4 0x6ecbc79 0x6ecbd7b 0x6ecbe73 0x6ecc080 0x6ecc266 0x6ecc5b5 0x6ecc959 0x6eccb89 0x6eccdf3 0x6ecd0c0 0x6ecd2de 0x6ecd396 0x6ecd45c 0x6ecd53d 0x5a2bf80 0x5a2c08a 0x5a2c146 0x5a2c202 0x5a2c2be 0x5a2c40b 0x5a2c596 0x5a2c957 0x5a48898 0xb875385 0xb88f61f 0xb93061d 0xb931c6b 0xb60e451 0xb60edcd 0x37cc91a 0x37cd473 0x366cd7a 0x2958f 0x2963f 0x3459444 2024-04-22T01:54:56.596 INFO:teuthology.orchestra.run.smithi107.stderr:kernel callstack: 2024-04-22T01:54:56.607 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:56,609 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T01:54:56.607 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:56,609 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-22T01:54:56.608 INFO:teuthology.orchestra.run.smithi107.stderr:DEBUG 2024-04-22 01:54:56,610 [shard 0:main] seastar - reactor::drain 2024-04-22T01:54:57.134 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:57.135 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-04-22T01:54:57.135 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-04-22T01:54:57.135 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-04-22T01:54:57.135 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:54:57.135 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T01:54:57.191 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:54:57.195 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-04-22 01:54:57,196 seastar - Reactor backend: linux-aio 2024-04-22T01:54:57.210 INFO:teuthology.orchestra.run.smithi107.stderr:==26779==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:55:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:55:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-04-22T01:55:01.838 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-04-22T01:55:01.839 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-04-22T01:55:01.839 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-04-22T01:55:01.839 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T01:55:01.839 INFO:teuthology.orchestra.run.smithi107.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:55:01.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T01:55:01.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T01:55:01.914 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T01:55:01.914 INFO:tasks.ceph:role: osd.2 2024-04-22T01:55:01.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T01:55:01.914 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T01:55:01.989 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:55:01.989 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:55:01.989 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:55:01.989 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:55:01.989 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:55:01.990 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:55:01.990 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:55:01.990 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:55:01.990 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:55:01.990 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:55:02.498 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-22T01:55:02.501 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-22T01:55:02.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-22T01:55:02.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T01:55:02.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-22T01:55:02.747 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-04-22T01:55:02.747 INFO:tasks.ceph:role: osd.3 2024-04-22T01:55:02.747 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T01:55:02.747 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T01:55:02.821 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:55:02.822 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:55:03.277 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-22T01:55:03.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-22T01:55:03.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-04-22T01:55:03.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-22T01:55:03.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:55:03.705 INFO:teuthology.orchestra.run.smithi150.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:55:03.706 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:03,708 seastar - Reactor backend: linux-aio 2024-04-22T01:55:03.717 INFO:teuthology.orchestra.run.smithi150.stderr:==26534==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:55:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:03,834 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T01:55:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:03,835 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-04-22T01:55:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:DEBUG 2024-04-22 01:55:03,835 [shard 0:main] seastar - reactor::drain 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr:Suppressions used: 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr: count bytes template 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr: 1 8 MallocExtension::Register 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:04.380 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-22T01:55:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:55:04.443 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:04,445 seastar - Reactor backend: linux-aio 2024-04-22T01:55:04.452 INFO:teuthology.orchestra.run.smithi150.stderr:==26532==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr:Suppressions used: 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr: count bytes template 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr: 1 8 MallocExtension::Register 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-22T01:55:09.082 INFO:teuthology.orchestra.run.smithi150.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:55:09.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T01:55:09.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:55:09.416 INFO:teuthology.orchestra.run.smithi150.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:55:09.417 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:09,419 seastar - Reactor backend: linux-aio 2024-04-22T01:55:09.422 INFO:teuthology.orchestra.run.smithi150.stderr:==26716==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:55:09.526 INFO:teuthology.orchestra.run.smithi150.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb98fc39 0xb862700 0xb863c7c 0xb863e92 0xb863fe8 0xb8644b1 0x3e6ef 0x113c06 0x4a0bd 0x35b7c 0x3629e 0x3680233 0x43c0903 0x5da2561 0x5d7e9a5 0x5bc3681 0x5bde745 0x6eb8f84 0x6eca757 0x6ecbbf4 0x6ecbc79 0x6ecbd7b 0x6ecbe73 0x6ecc080 0x6ecc266 0x6ecc5b5 0x6ecc959 0x6eccb89 0x6eccdf3 0x6ecd0c0 0x6ecd2de 0x6ecd396 0x6ecd45c 0x6ecd53d 0x5a2bf80 0x5a2c08a 0x5a2c146 0x5a2c202 0x5a2c2be 0x5a2c40b 0x5a2c596 0x5a2c957 0x5a48898 0xb875385 0xb88f61f 0xb93061d 0xb931c6b 0xb60e451 0xb60edcd 0x37cc91a 0x37cd473 0x366cd7a 0x2958f 0x2963f 0x3459444 2024-04-22T01:55:09.526 INFO:teuthology.orchestra.run.smithi150.stderr:kernel callstack: 2024-04-22T01:55:09.537 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:09,539 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-22T01:55:09.537 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:09,539 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-04-22T01:55:09.538 INFO:teuthology.orchestra.run.smithi150.stderr:DEBUG 2024-04-22 01:55:09,540 [shard 0:main] seastar - reactor::drain 2024-04-22T01:55:10.073 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:10.074 INFO:teuthology.orchestra.run.smithi150.stderr:Suppressions used: 2024-04-22T01:55:10.074 INFO:teuthology.orchestra.run.smithi150.stderr: count bytes template 2024-04-22T01:55:10.074 INFO:teuthology.orchestra.run.smithi150.stderr: 1 8 MallocExtension::Register 2024-04-22T01:55:10.074 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:10.074 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-22T01:55:10.135 INFO:teuthology.orchestra.run.smithi150.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:55:10.138 INFO:teuthology.orchestra.run.smithi150.stderr:INFO 2024-04-22 01:55:10,140 seastar - Reactor backend: linux-aio 2024-04-22T01:55:10.153 INFO:teuthology.orchestra.run.smithi150.stderr:==26714==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:55:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:Suppressions used: 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stderr: count bytes template 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stderr: 1 8 MallocExtension::Register 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:----------------------------------------------------- 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-22T01:55:14.790 INFO:teuthology.orchestra.run.smithi150.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:55:14.812 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-22T01:55:14.835 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T01:55:14.835 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:55:14.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T01:55:14.864 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:55:14.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T01:55:14.932 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:14.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T01:55:14.956 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:14.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T01:55:15.019 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:15.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-22T01:55:15.083 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:55:15.083 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T01:55:15.101 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:15.102 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-22T01:55:15.138 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T01:55:15.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBrwyVm5u62JxAAAFON0qXXWF/5TzjvJGJUxw== 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBxwyVmKu6kFBAAN1M3YQJtzuYsqyEBOxT7UA== 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBowyVmkp3GKhAAUm9ossR7XIbsnE6cBcGdTQ== 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQB4wyVmbCpYIxAASgs8W+Op0O+0witncPxM1A== 2024-04-22T01:55:15.168 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2024-04-22T01:55:15.169 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQB+wyVme0GDERAA5stSD2HjcWOflqwkua03+A== 2024-04-22T01:55:15.169 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-04-22T01:55:15.169 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBowyVmTgLLLRAAHFcWvj2hE0gpl/7dbG44Aw== 2024-04-22T01:55:15.169 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2024-04-22T01:55:15.169 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBowyVmsSJmMRAAh49b0jK2qUMFjE1BV4Tmmg== 2024-04-22T01:55:15.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:55:15.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:55:15.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:55:15.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:55:15.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:55:15.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:55:15.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:55:15.636 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T01:55:15.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T01:55:15.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T01:55:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:15.766+0000 7fde33ea3a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:15.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:15.766+0000 7fde33ea3a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:15.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T01:55:15.801 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:55:15.857 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T01:55:15.858 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T01:55:15.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T01:55:15.900 INFO:tasks.ceph.mon.a:Started 2024-04-22T01:55:15.900 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T01:55:15.900 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T01:55:15.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T01:55:15.903 INFO:tasks.ceph.mgr.x:Started 2024-04-22T01:55:15.903 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T01:55:15.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T01:55:15.980 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T01:55:15.981+0000 7f6b22c70a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:15.980 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T01:55:15.981+0000 7f6b22c70a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:15.980 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T01:55:15.981+0000 7f6b22c70a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:15.994 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:15.994+0000 7f398862d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:16.005+0000 7f18aa00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:16.014+0000 7f18aa00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.014+0000 7f398862d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.014 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.014+0000 7f398862d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-04-22T01:55:16.096 INFO:tasks.ceph:check_enable_crimson: True 2024-04-22T01:55:16.096 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-22T01:55:16.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-22T01:55:16.116 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.116+0000 7f398862d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:55:16.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:16.195+0000 7f6699e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:16.204+0000 7f6699e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:16.261 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.262+0000 7f398862d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:55:16.281 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T01:55:16.282+0000 7f6b199f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-22T01:55:16.281 INFO:tasks.ceph.mon.a.smithi107.stderr:Please be aware that this feature is experimental, untested, 2024-04-22T01:55:16.281 INFO:tasks.ceph.mon.a.smithi107.stderr:unsupported, and may result in data corruption, data loss, 2024-04-22T01:55:16.282 INFO:tasks.ceph.mon.a.smithi107.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-22T01:55:16.282 INFO:tasks.ceph.mon.a.smithi107.stderr:feature with important data. 2024-04-22T01:55:16.282 INFO:tasks.ceph.mon.a.smithi107.stderr: 2024-04-22T01:55:16.314 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.314+0000 7f398862d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:55:16.562 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.563+0000 7f398862d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:55:16.606 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:16.607+0000 7f398862d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:55:17.099 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T01:55:17.099 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:55:17.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T01:55:17.124 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-04-22T01:55:17.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T01:55:17.190 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:17.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T01:55:17.214 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-22T01:55:17.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-22T01:55:17.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 08b6bf71-e816-48c5-b073-b7ec9b5acd4e 0 2024-04-22T01:55:17.302 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.303+0000 7f398862d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:55:17.396 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.397+0000 7f398862d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:55:17.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.403+0000 7f32bbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.413+0000 7f32bbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.441 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.442+0000 7f398862d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:55:17.491 INFO:teuthology.orchestra.run.smithi150.stdout:0 2024-04-22T01:55:17.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new dbd2b417-bbcf-440d-ad42-3232cbb2b5a8 1 2024-04-22T01:55:17.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.605+0000 7ff2d626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.614+0000 7ff2d626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.618 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.618+0000 7f398862d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:55:17.667 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.668+0000 7f398862d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:55:17.693 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-22T01:55:17.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 0fc13a13-783d-4b86-870d-3d83caa6e82d 2 2024-04-22T01:55:17.723 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.724+0000 7f398862d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:55:17.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.806+0000 7fe32bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:17.815+0000 7fe32bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:17.842 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:17.843+0000 7f398862d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:55:17.894 INFO:teuthology.orchestra.run.smithi150.stdout:2 2024-04-22T01:55:17.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 2c6e979e-5a68-4bff-8304-ef2133dfaf2a 3 2024-04-22T01:55:18.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:18.047+0000 7fd174820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:18.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-22T01:55:18.056+0000 7fd174820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:18.136 INFO:teuthology.orchestra.run.smithi150.stdout:3 2024-04-22T01:55:18.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-22T01:55:18.182 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:55:18.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:55:18.225 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:55:18.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-22T01:55:18.233 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:18.233+0000 7f398862d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:55:18.255 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T01:55:18.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T01:55:18.297 INFO:tasks.ceph.osd.1:Started 2024-04-22T01:55:18.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-22T01:55:18.327 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T01:55:18.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T01:55:18.370 INFO:tasks.ceph.osd.2:Started 2024-04-22T01:55:18.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-04-22T01:55:18.399 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T01:55:18.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-22T01:55:18.442 INFO:tasks.ceph.osd.3:Started 2024-04-22T01:55:18.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:55:18.510 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:18.511+0000 7f398862d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:55:18.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:18.553+0000 7f8b65954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:18.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:18.561+0000 7f8b65954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:18.571 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:18.571+0000 7f398862d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:55:18.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:18.629 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":7,"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","created":"2024-04-22T01:55:16.002883+0000","modified":"2024-04-22T01:55:18.134264+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:55:18.134264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"08b6bf71-e816-48c5-b073-b7ec9b5acd4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dbd2b417-bbcf-440d-ad42-3232cbb2b5a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0fc13a13-783d-4b86-870d-3d83caa6e82d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c6e979e-5a68-4bff-8304-ef2133dfaf2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:55:18.640 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T01:55:18.641 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T01:55:18.700 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:18.701+0000 7f398862d000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:55:18.994 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:18.995+0000 7f398862d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:55:19.045 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:55:19.048 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.048+0000 7f398862d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:55:19.150 INFO:teuthology.misc.health.smithi107.stderr:2024-04-22T01:55:19.147+0000 7f708015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:19.154 INFO:teuthology.misc.health.smithi107.stderr:2024-04-22T01:55:19.155+0000 7f708015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:19.220 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.221+0000 7f398862d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:55:19.223 INFO:teuthology.misc.health.smithi107.stdout: 2024-04-22T01:55:19.223 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":7,"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","created":"2024-04-22T01:55:16.002883+0000","modified":"2024-04-22T01:55:18.134264+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:55:18.134264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"08b6bf71-e816-48c5-b073-b7ec9b5acd4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dbd2b417-bbcf-440d-ad42-3232cbb2b5a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0fc13a13-783d-4b86-870d-3d83caa6e82d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c6e979e-5a68-4bff-8304-ef2133dfaf2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:55:19.233 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-22T01:55:19.361 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.361+0000 7f398862d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:55:19.415 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.415+0000 7f398862d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:55:19.490 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.491+0000 7f398862d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:55:19.538 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:19.539+0000 7f398862d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:55:20.033 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:20.034+0000 7f398862d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:55:20.168 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:20.168+0000 7f398862d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:55:21.180 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T01:55:21.180+0000 7f3927390640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-22T01:55:25.638 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:55:25.741 INFO:teuthology.misc.health.smithi107.stderr:2024-04-22T01:55:25.738+0000 7f58eb6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:25.745 INFO:teuthology.misc.health.smithi107.stderr:2024-04-22T01:55:25.746+0000 7f58eb6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:25.947 INFO:teuthology.misc.health.smithi107.stdout: 2024-04-22T01:55:25.947 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":11,"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","created":"2024-04-22T01:55:16.002883+0000","modified":"2024-04-22T01:55:25.246088+0000","last_up_change":"2024-04-22T01:55:23.181753+0000","last_in_change":"2024-04-22T01:55:18.134264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:55:24.194959+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"08b6bf71-e816-48c5-b073-b7ec9b5acd4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":464935848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":464935848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":464935848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":464935848}]},"public_addr":"172.21.15.107:6801/464935848","cluster_addr":"172.21.15.107:6802/464935848","heartbeat_back_addr":"172.21.15.107:6806/464935848","heartbeat_front_addr":"172.21.15.107:6803/464935848","state":["exists","up"]},{"osd":1,"uuid":"dbd2b417-bbcf-440d-ad42-3232cbb2b5a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3294686425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3294686425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3294686425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3294686425}]},"public_addr":"172.21.15.107:6805/3294686425","cluster_addr":"172.21.15.107:6804/3294686425","heartbeat_back_addr":"172.21.15.107:6809/3294686425","heartbeat_front_addr":"172.21.15.107:6808/3294686425","state":["exists","up"]},{"osd":2,"uuid":"0fc13a13-783d-4b86-870d-3d83caa6e82d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":3900725868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3900725868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":3900725868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3900725868}]},"public_addr":"172.21.15.150:6803/3900725868","cluster_addr":"172.21.15.150:6802/3900725868","heartbeat_back_addr":"172.21.15.150:6811/3900725868","heartbeat_front_addr":"172.21.15.150:6810/3900725868","state":["exists","up"]},{"osd":3,"uuid":"2c6e979e-5a68-4bff-8304-ef2133dfaf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3017050861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":3017050861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3017050861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3017050861}]},"public_addr":"172.21.15.150:6804/3017050861","cluster_addr":"172.21.15.150:6805/3017050861","heartbeat_back_addr":"172.21.15.150:6809/3017050861","heartbeat_front_addr":"172.21.15.150:6808/3017050861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:55:25.959 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-22T01:55:25.959 INFO:tasks.ceph:Creating RBD pool 2024-04-22T01:55:25.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T01:55:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:26.066+0000 7f45590ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:26.078+0000 7f45590ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:27.253 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2024-04-22T01:55:27.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T01:55:27.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:27.389+0000 7fd2fcd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:27.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:27.402+0000 7fd2fcd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T01:55:28.278 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T01:55:28.279 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T01:55:28.279 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T01:55:28.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T01:55:28.279 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T01:55:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:28.418+0000 7f16f859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:28.425+0000 7f16f859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:28.673 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":563203308},{"type":"v1","addr":"172.21.15.150:6801","nonce":563203308}]},"active_addr":"172.21.15.150:6801/563203308","active_change":"2024-04-22T01:55:20.171417+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1587319877}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1799612067}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3132350671}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":467578356}]}]} 2024-04-22T01:55:28.675 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T01:55:28.675 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T01:55:28.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:55:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:28.778+0000 7fea80a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:28.786+0000 7fea80a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:28.972 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:28.973 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":14,"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","created":"2024-04-22T01:55:16.002883+0000","modified":"2024-04-22T01:55:28.256028+0000","last_up_change":"2024-04-22T01:55:23.181753+0000","last_in_change":"2024-04-22T01:55:18.134264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:55:24.194959+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T01:55:26.331416+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"08b6bf71-e816-48c5-b073-b7ec9b5acd4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":464935848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":464935848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":464935848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":464935848}]},"public_addr":"172.21.15.107:6801/464935848","cluster_addr":"172.21.15.107:6802/464935848","heartbeat_back_addr":"172.21.15.107:6806/464935848","heartbeat_front_addr":"172.21.15.107:6803/464935848","state":["exists","up"]},{"osd":1,"uuid":"dbd2b417-bbcf-440d-ad42-3232cbb2b5a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3294686425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3294686425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3294686425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3294686425}]},"public_addr":"172.21.15.107:6805/3294686425","cluster_addr":"172.21.15.107:6804/3294686425","heartbeat_back_addr":"172.21.15.107:6809/3294686425","heartbeat_front_addr":"172.21.15.107:6808/3294686425","state":["exists","up"]},{"osd":2,"uuid":"0fc13a13-783d-4b86-870d-3d83caa6e82d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":3900725868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3900725868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":3900725868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3900725868}]},"public_addr":"172.21.15.150:6803/3900725868","cluster_addr":"172.21.15.150:6802/3900725868","heartbeat_back_addr":"172.21.15.150:6811/3900725868","heartbeat_front_addr":"172.21.15.150:6810/3900725868","state":["exists","up"]},{"osd":3,"uuid":"2c6e979e-5a68-4bff-8304-ef2133dfaf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3017050861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":3017050861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3017050861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3017050861}]},"public_addr":"172.21.15.150:6804/3017050861","cluster_addr":"172.21.15.150:6805/3017050861","heartbeat_back_addr":"172.21.15.150:6809/3017050861","heartbeat_front_addr":"172.21.15.150:6808/3017050861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:55:28.986 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T01:55:28.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:55:29.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.090+0000 7f4b2d1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.098+0000 7f4b2d1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.286 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:29.287 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":15,"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","created":"2024-04-22T01:55:16.002883+0000","modified":"2024-04-22T01:55:29.261878+0000","last_up_change":"2024-04-22T01:55:23.181753+0000","last_in_change":"2024-04-22T01:55:18.134264+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:55:24.194959+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T01:55:26.331416+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"08b6bf71-e816-48c5-b073-b7ec9b5acd4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6801","nonce":464935848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":464935848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":464935848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":464935848}]},"public_addr":"172.21.15.107:6801/464935848","cluster_addr":"172.21.15.107:6802/464935848","heartbeat_back_addr":"172.21.15.107:6806/464935848","heartbeat_front_addr":"172.21.15.107:6803/464935848","state":["exists","up"]},{"osd":1,"uuid":"dbd2b417-bbcf-440d-ad42-3232cbb2b5a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3294686425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3294686425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3294686425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3294686425}]},"public_addr":"172.21.15.107:6805/3294686425","cluster_addr":"172.21.15.107:6804/3294686425","heartbeat_back_addr":"172.21.15.107:6809/3294686425","heartbeat_front_addr":"172.21.15.107:6808/3294686425","state":["exists","up"]},{"osd":2,"uuid":"0fc13a13-783d-4b86-870d-3d83caa6e82d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6803","nonce":3900725868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":3900725868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6811","nonce":3900725868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":3900725868}]},"public_addr":"172.21.15.150:6803/3900725868","cluster_addr":"172.21.15.150:6802/3900725868","heartbeat_back_addr":"172.21.15.150:6811/3900725868","heartbeat_front_addr":"172.21.15.150:6810/3900725868","state":["exists","up"]},{"osd":3,"uuid":"2c6e979e-5a68-4bff-8304-ef2133dfaf2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":3017050861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6805","nonce":3017050861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6809","nonce":3017050861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":3017050861}]},"public_addr":"172.21.15.150:6804/3017050861","cluster_addr":"172.21.15.150:6805/3017050861","heartbeat_back_addr":"172.21.15.150:6809/3017050861","heartbeat_front_addr":"172.21.15.150:6808/3017050861","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:55:29.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T01:55:29.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T01:55:29.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T01:55:29.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-22T01:55:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.412+0000 7f332c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.412+0000 7fa4270ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.421+0000 7f332c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.412+0000 7f39eff1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.422+0000 7fa4270ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.412+0000 7fbe2846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.427+0000 7f39eff1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.428+0000 7fbe2846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.458 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:29.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T01:55:29.459 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:29.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:55:29.460 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:29.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:55:29.463 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:29.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:55:29.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.571+0000 7f0264d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.584+0000 7f0264d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.615+0000 7f341c8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.619+0000 7f0de5824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.619+0000 7f4a256ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.625+0000 7f341c8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.631+0000 7f4a256ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:29.631+0000 7f0de5824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:29.801 INFO:teuthology.orchestra.run.smithi107.stdout:38654705669 2024-04-22T01:55:29.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.3 2024-04-22T01:55:29.827 INFO:teuthology.orchestra.run.smithi107.stdout:38654705669 2024-04-22T01:55:29.832 INFO:teuthology.orchestra.run.smithi107.stdout:38654705669 2024-04-22T01:55:29.840 INFO:teuthology.orchestra.run.smithi107.stdout:38654705669 2024-04-22T01:55:29.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.0 2024-04-22T01:55:29.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2024-04-22T01:55:29.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.1 2024-04-22T01:55:30.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-22T01:55:30.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:55:30.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:55:30.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:55:30.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:30.950+0000 7fcc9c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:30.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:30.967+0000 7fcc9c35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:30.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:30.972+0000 7f0b4799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:30.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:30.982+0000 7f0b4799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:30.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:30.977+0000 7f70c7ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.000+0000 7f70c7ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.003+0000 7f31f8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.013+0000 7f31f8a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.210 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:31.212 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:31.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.1 2024-04-22T01:55:31.225 DEBUG:teuthology.parallel:result is None 2024-04-22T01:55:31.231 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2024-04-22T01:55:31.231 DEBUG:teuthology.parallel:result is None 2024-04-22T01:55:31.233 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:31.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.0 2024-04-22T01:55:31.247 DEBUG:teuthology.parallel:result is None 2024-04-22T01:55:31.299 INFO:teuthology.orchestra.run.smithi107.stdout:38654705670 2024-04-22T01:55:31.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.3 2024-04-22T01:55:31.313 DEBUG:teuthology.parallel:result is None 2024-04-22T01:55:31.313 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T01:55:31.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:55:31.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.417+0000 7f779c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.425+0000 7f779c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.616 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:55:31.628 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-22T01:55:30.179042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107968,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379392,"statfs":{"total":386547056640,"available":386436497408,"internally_reserved":0,"allocated":659456,"data_stored":1890176,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.969867"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.399731+0000","last_change":"2024-04-22T01:55:28.401188+0000","last_active":"2024-04-22T01:55:28.399731+0000","last_peered":"2024-04-22T01:55:28.399731+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.394657+0000","last_became_peered":"2024-04-22T01:55:28.394657+0000","last_unstale":"2024-04-22T01:55:28.399731+0000","last_undegraded":"2024-04-22T01:55:28.399731+0000","last_fullsized":"2024-04-22T01:55:28.399731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.283939+0000","last_change":"2024-04-22T01:55:28.353361+0000","last_active":"2024-04-22T01:55:29.283939+0000","last_peered":"2024-04-22T01:55:29.283939+0000","last_clean":"2024-04-22T01:55:29.283939+0000","last_became_active":"2024-04-22T01:55:28.345178+0000","last_became_peered":"2024-04-22T01:55:28.345178+0000","last_unstale":"2024-04-22T01:55:29.283939+0000","last_undegraded":"2024-04-22T01:55:29.283939+0000","last_fullsized":"2024-04-22T01:55:29.283939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.343387+0000","last_change":"2024-04-22T01:55:28.344798+0000","last_active":"2024-04-22T01:55:28.343387+0000","last_peered":"2024-04-22T01:55:28.343387+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.338687+0000","last_became_peered":"2024-04-22T01:55:28.338687+0000","last_unstale":"2024-04-22T01:55:28.343387+0000","last_undegraded":"2024-04-22T01:55:28.343387+0000","last_fullsized":"2024-04-22T01:55:28.343387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.288760+0000","last_change":"2024-04-22T01:55:28.349864+0000","last_active":"2024-04-22T01:55:29.288760+0000","last_peered":"2024-04-22T01:55:29.288760+0000","last_clean":"2024-04-22T01:55:29.288760+0000","last_became_active":"2024-04-22T01:55:28.345331+0000","last_became_peered":"2024-04-22T01:55:28.345331+0000","last_unstale":"2024-04-22T01:55:29.288760+0000","last_undegraded":"2024-04-22T01:55:29.288760+0000","last_fullsized":"2024-04-22T01:55:29.288760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.346145+0000","last_change":"2024-04-22T01:55:28.347626+0000","last_active":"2024-04-22T01:55:28.346145+0000","last_peered":"2024-04-22T01:55:28.346145+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.341199+0000","last_became_peered":"2024-04-22T01:55:28.341199+0000","last_unstale":"2024-04-22T01:55:28.346145+0000","last_undegraded":"2024-04-22T01:55:28.346145+0000","last_fullsized":"2024-04-22T01:55:28.346145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.363244+0000","last_change":"2024-04-22T01:55:28.364694+0000","last_active":"2024-04-22T01:55:28.363244+0000","last_peered":"2024-04-22T01:55:28.363244+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.355830+0000","last_became_peered":"2024-04-22T01:55:28.355830+0000","last_unstale":"2024-04-22T01:55:28.363244+0000","last_undegraded":"2024-04-22T01:55:28.363244+0000","last_fullsized":"2024-04-22T01:55:28.363244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.355667+0000","last_change":"2024-04-22T01:55:28.356547+0000","last_active":"2024-04-22T01:55:28.355667+0000","last_peered":"2024-04-22T01:55:28.355667+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.351325+0000","last_became_peered":"2024-04-22T01:55:28.351325+0000","last_unstale":"2024-04-22T01:55:28.355667+0000","last_undegraded":"2024-04-22T01:55:28.355667+0000","last_fullsized":"2024-04-22T01:55:28.355667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.388358+0000","last_change":"2024-04-22T01:55:28.389444+0000","last_active":"2024-04-22T01:55:28.388358+0000","last_peered":"2024-04-22T01:55:28.388358+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.384392+0000","last_became_peered":"2024-04-22T01:55:28.384392+0000","last_unstale":"2024-04-22T01:55:28.388358+0000","last_undegraded":"2024-04-22T01:55:28.388358+0000","last_fullsized":"2024-04-22T01:55:28.388358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.275472+0000","last_change":"2024-04-22T01:55:25.303769+0000","last_active":"2024-04-22T01:55:28.275472+0000","last_peered":"2024-04-22T01:55:28.275472+0000","last_clean":"2024-04-22T01:55:28.275472+0000","last_became_active":"2024-04-22T01:55:25.298462+0000","last_became_peered":"2024-04-22T01:55:25.298462+0000","last_unstale":"2024-04-22T01:55:28.275472+0000","last_undegraded":"2024-04-22T01:55:28.275472+0000","last_fullsized":"2024-04-22T01:55:28.275472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:55:31.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:55:31.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.733+0000 7f5460064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:31.741+0000 7f5460064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:31.925 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:31.925 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:55:31.937 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-04-22T01:55:30.179042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107968,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379392,"statfs":{"total":386547056640,"available":386436497408,"internally_reserved":0,"allocated":659456,"data_stored":1890176,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.969867"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.399731+0000","last_change":"2024-04-22T01:55:28.401188+0000","last_active":"2024-04-22T01:55:28.399731+0000","last_peered":"2024-04-22T01:55:28.399731+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.394657+0000","last_became_peered":"2024-04-22T01:55:28.394657+0000","last_unstale":"2024-04-22T01:55:28.399731+0000","last_undegraded":"2024-04-22T01:55:28.399731+0000","last_fullsized":"2024-04-22T01:55:28.399731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.283939+0000","last_change":"2024-04-22T01:55:28.353361+0000","last_active":"2024-04-22T01:55:29.283939+0000","last_peered":"2024-04-22T01:55:29.283939+0000","last_clean":"2024-04-22T01:55:29.283939+0000","last_became_active":"2024-04-22T01:55:28.345178+0000","last_became_peered":"2024-04-22T01:55:28.345178+0000","last_unstale":"2024-04-22T01:55:29.283939+0000","last_undegraded":"2024-04-22T01:55:29.283939+0000","last_fullsized":"2024-04-22T01:55:29.283939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.343387+0000","last_change":"2024-04-22T01:55:28.344798+0000","last_active":"2024-04-22T01:55:28.343387+0000","last_peered":"2024-04-22T01:55:28.343387+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.338687+0000","last_became_peered":"2024-04-22T01:55:28.338687+0000","last_unstale":"2024-04-22T01:55:28.343387+0000","last_undegraded":"2024-04-22T01:55:28.343387+0000","last_fullsized":"2024-04-22T01:55:28.343387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.288760+0000","last_change":"2024-04-22T01:55:28.349864+0000","last_active":"2024-04-22T01:55:29.288760+0000","last_peered":"2024-04-22T01:55:29.288760+0000","last_clean":"2024-04-22T01:55:29.288760+0000","last_became_active":"2024-04-22T01:55:28.345331+0000","last_became_peered":"2024-04-22T01:55:28.345331+0000","last_unstale":"2024-04-22T01:55:29.288760+0000","last_undegraded":"2024-04-22T01:55:29.288760+0000","last_fullsized":"2024-04-22T01:55:29.288760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.346145+0000","last_change":"2024-04-22T01:55:28.347626+0000","last_active":"2024-04-22T01:55:28.346145+0000","last_peered":"2024-04-22T01:55:28.346145+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.341199+0000","last_became_peered":"2024-04-22T01:55:28.341199+0000","last_unstale":"2024-04-22T01:55:28.346145+0000","last_undegraded":"2024-04-22T01:55:28.346145+0000","last_fullsized":"2024-04-22T01:55:28.346145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.363244+0000","last_change":"2024-04-22T01:55:28.364694+0000","last_active":"2024-04-22T01:55:28.363244+0000","last_peered":"2024-04-22T01:55:28.363244+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.355830+0000","last_became_peered":"2024-04-22T01:55:28.355830+0000","last_unstale":"2024-04-22T01:55:28.363244+0000","last_undegraded":"2024-04-22T01:55:28.363244+0000","last_fullsized":"2024-04-22T01:55:28.363244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.355667+0000","last_change":"2024-04-22T01:55:28.356547+0000","last_active":"2024-04-22T01:55:28.355667+0000","last_peered":"2024-04-22T01:55:28.355667+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.351325+0000","last_became_peered":"2024-04-22T01:55:28.351325+0000","last_unstale":"2024-04-22T01:55:28.355667+0000","last_undegraded":"2024-04-22T01:55:28.355667+0000","last_fullsized":"2024-04-22T01:55:28.355667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.388358+0000","last_change":"2024-04-22T01:55:28.389444+0000","last_active":"2024-04-22T01:55:28.388358+0000","last_peered":"2024-04-22T01:55:28.388358+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.384392+0000","last_became_peered":"2024-04-22T01:55:28.384392+0000","last_unstale":"2024-04-22T01:55:28.388358+0000","last_undegraded":"2024-04-22T01:55:28.388358+0000","last_fullsized":"2024-04-22T01:55:28.388358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.275472+0000","last_change":"2024-04-22T01:55:25.303769+0000","last_active":"2024-04-22T01:55:28.275472+0000","last_peered":"2024-04-22T01:55:28.275472+0000","last_clean":"2024-04-22T01:55:28.275472+0000","last_became_active":"2024-04-22T01:55:25.298462+0000","last_became_peered":"2024-04-22T01:55:25.298462+0000","last_unstale":"2024-04-22T01:55:28.275472+0000","last_undegraded":"2024-04-22T01:55:28.275472+0000","last_fullsized":"2024-04-22T01:55:28.275472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:55:31.938 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T01:55:31.938 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T01:55:31.938 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T01:55:31.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T01:55:32.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.042+0000 7fb51d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.049+0000 7fb51d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.275 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:32.276 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T01:55:32.289 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T01:55:32.290 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-22T01:55:32.300 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-22T01:55:32.300 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}} 2024-04-22T01:55:32.300 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-22T01:55:32.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-22T01:55:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.404+0000 7f5229b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.415+0000 7f5229b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.617 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-22T01:55:32.617 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T01:55:32.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:55:32.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.720+0000 7efd10e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:32.728+0000 7efd10e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:32.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:32.913 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:55:32.924 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-22T01:55:32.179506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107968,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379392,"statfs":{"total":386547056640,"available":386436497408,"internally_reserved":0,"allocated":659456,"data_stored":1890176,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970331"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.399731+0000","last_change":"2024-04-22T01:55:28.401188+0000","last_active":"2024-04-22T01:55:28.399731+0000","last_peered":"2024-04-22T01:55:28.399731+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.394657+0000","last_became_peered":"2024-04-22T01:55:28.394657+0000","last_unstale":"2024-04-22T01:55:28.399731+0000","last_undegraded":"2024-04-22T01:55:28.399731+0000","last_fullsized":"2024-04-22T01:55:28.399731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.283939+0000","last_change":"2024-04-22T01:55:28.353361+0000","last_active":"2024-04-22T01:55:29.283939+0000","last_peered":"2024-04-22T01:55:29.283939+0000","last_clean":"2024-04-22T01:55:29.283939+0000","last_became_active":"2024-04-22T01:55:28.345178+0000","last_became_peered":"2024-04-22T01:55:28.345178+0000","last_unstale":"2024-04-22T01:55:29.283939+0000","last_undegraded":"2024-04-22T01:55:29.283939+0000","last_fullsized":"2024-04-22T01:55:29.283939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.343387+0000","last_change":"2024-04-22T01:55:28.344798+0000","last_active":"2024-04-22T01:55:28.343387+0000","last_peered":"2024-04-22T01:55:28.343387+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.338687+0000","last_became_peered":"2024-04-22T01:55:28.338687+0000","last_unstale":"2024-04-22T01:55:28.343387+0000","last_undegraded":"2024-04-22T01:55:28.343387+0000","last_fullsized":"2024-04-22T01:55:28.343387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.288760+0000","last_change":"2024-04-22T01:55:28.349864+0000","last_active":"2024-04-22T01:55:29.288760+0000","last_peered":"2024-04-22T01:55:29.288760+0000","last_clean":"2024-04-22T01:55:29.288760+0000","last_became_active":"2024-04-22T01:55:28.345331+0000","last_became_peered":"2024-04-22T01:55:28.345331+0000","last_unstale":"2024-04-22T01:55:29.288760+0000","last_undegraded":"2024-04-22T01:55:29.288760+0000","last_fullsized":"2024-04-22T01:55:29.288760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.346145+0000","last_change":"2024-04-22T01:55:28.347626+0000","last_active":"2024-04-22T01:55:28.346145+0000","last_peered":"2024-04-22T01:55:28.346145+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.341199+0000","last_became_peered":"2024-04-22T01:55:28.341199+0000","last_unstale":"2024-04-22T01:55:28.346145+0000","last_undegraded":"2024-04-22T01:55:28.346145+0000","last_fullsized":"2024-04-22T01:55:28.346145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.363244+0000","last_change":"2024-04-22T01:55:28.364694+0000","last_active":"2024-04-22T01:55:28.363244+0000","last_peered":"2024-04-22T01:55:28.363244+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.355830+0000","last_became_peered":"2024-04-22T01:55:28.355830+0000","last_unstale":"2024-04-22T01:55:28.363244+0000","last_undegraded":"2024-04-22T01:55:28.363244+0000","last_fullsized":"2024-04-22T01:55:28.363244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.355667+0000","last_change":"2024-04-22T01:55:28.356547+0000","last_active":"2024-04-22T01:55:28.355667+0000","last_peered":"2024-04-22T01:55:28.355667+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.351325+0000","last_became_peered":"2024-04-22T01:55:28.351325+0000","last_unstale":"2024-04-22T01:55:28.355667+0000","last_undegraded":"2024-04-22T01:55:28.355667+0000","last_fullsized":"2024-04-22T01:55:28.355667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.388358+0000","last_change":"2024-04-22T01:55:28.389444+0000","last_active":"2024-04-22T01:55:28.388358+0000","last_peered":"2024-04-22T01:55:28.388358+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.384392+0000","last_became_peered":"2024-04-22T01:55:28.384392+0000","last_unstale":"2024-04-22T01:55:28.388358+0000","last_undegraded":"2024-04-22T01:55:28.388358+0000","last_fullsized":"2024-04-22T01:55:28.388358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.275472+0000","last_change":"2024-04-22T01:55:25.303769+0000","last_active":"2024-04-22T01:55:28.275472+0000","last_peered":"2024-04-22T01:55:28.275472+0000","last_clean":"2024-04-22T01:55:28.275472+0000","last_became_active":"2024-04-22T01:55:25.298462+0000","last_became_peered":"2024-04-22T01:55:25.298462+0000","last_unstale":"2024-04-22T01:55:28.275472+0000","last_undegraded":"2024-04-22T01:55:28.275472+0000","last_fullsized":"2024-04-22T01:55:28.275472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:55:32.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:55:33.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.029+0000 7f0f9491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.037+0000 7f0f9491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.222 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:55:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:55:33.235 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-22T01:55:32.179506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107968,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379392,"statfs":{"total":386547056640,"available":386436497408,"internally_reserved":0,"allocated":659456,"data_stored":1890176,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970331"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.399731+0000","last_change":"2024-04-22T01:55:28.401188+0000","last_active":"2024-04-22T01:55:28.399731+0000","last_peered":"2024-04-22T01:55:28.399731+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.394657+0000","last_became_peered":"2024-04-22T01:55:28.394657+0000","last_unstale":"2024-04-22T01:55:28.399731+0000","last_undegraded":"2024-04-22T01:55:28.399731+0000","last_fullsized":"2024-04-22T01:55:28.399731+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.283939+0000","last_change":"2024-04-22T01:55:28.353361+0000","last_active":"2024-04-22T01:55:29.283939+0000","last_peered":"2024-04-22T01:55:29.283939+0000","last_clean":"2024-04-22T01:55:29.283939+0000","last_became_active":"2024-04-22T01:55:28.345178+0000","last_became_peered":"2024-04-22T01:55:28.345178+0000","last_unstale":"2024-04-22T01:55:29.283939+0000","last_undegraded":"2024-04-22T01:55:29.283939+0000","last_fullsized":"2024-04-22T01:55:29.283939+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.343387+0000","last_change":"2024-04-22T01:55:28.344798+0000","last_active":"2024-04-22T01:55:28.343387+0000","last_peered":"2024-04-22T01:55:28.343387+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.338687+0000","last_became_peered":"2024-04-22T01:55:28.338687+0000","last_unstale":"2024-04-22T01:55:28.343387+0000","last_undegraded":"2024-04-22T01:55:28.343387+0000","last_fullsized":"2024-04-22T01:55:28.343387+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-22T01:55:29.288760+0000","last_change":"2024-04-22T01:55:28.349864+0000","last_active":"2024-04-22T01:55:29.288760+0000","last_peered":"2024-04-22T01:55:29.288760+0000","last_clean":"2024-04-22T01:55:29.288760+0000","last_became_active":"2024-04-22T01:55:28.345331+0000","last_became_peered":"2024-04-22T01:55:28.345331+0000","last_unstale":"2024-04-22T01:55:29.288760+0000","last_undegraded":"2024-04-22T01:55:29.288760+0000","last_fullsized":"2024-04-22T01:55:29.288760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.346145+0000","last_change":"2024-04-22T01:55:28.347626+0000","last_active":"2024-04-22T01:55:28.346145+0000","last_peered":"2024-04-22T01:55:28.346145+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.341199+0000","last_became_peered":"2024-04-22T01:55:28.341199+0000","last_unstale":"2024-04-22T01:55:28.346145+0000","last_undegraded":"2024-04-22T01:55:28.346145+0000","last_fullsized":"2024-04-22T01:55:28.346145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.363244+0000","last_change":"2024-04-22T01:55:28.364694+0000","last_active":"2024-04-22T01:55:28.363244+0000","last_peered":"2024-04-22T01:55:28.363244+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.355830+0000","last_became_peered":"2024-04-22T01:55:28.355830+0000","last_unstale":"2024-04-22T01:55:28.363244+0000","last_undegraded":"2024-04-22T01:55:28.363244+0000","last_fullsized":"2024-04-22T01:55:28.363244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.355667+0000","last_change":"2024-04-22T01:55:28.356547+0000","last_active":"2024-04-22T01:55:28.355667+0000","last_peered":"2024-04-22T01:55:28.355667+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.351325+0000","last_became_peered":"2024-04-22T01:55:28.351325+0000","last_unstale":"2024-04-22T01:55:28.355667+0000","last_undegraded":"2024-04-22T01:55:28.355667+0000","last_fullsized":"2024-04-22T01:55:28.355667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.388358+0000","last_change":"2024-04-22T01:55:28.389444+0000","last_active":"2024-04-22T01:55:28.388358+0000","last_peered":"2024-04-22T01:55:28.388358+0000","last_clean":"2024-04-22T01:55:27.252369+0000","last_became_active":"2024-04-22T01:55:28.384392+0000","last_became_peered":"2024-04-22T01:55:28.384392+0000","last_unstale":"2024-04-22T01:55:28.388358+0000","last_undegraded":"2024-04-22T01:55:28.388358+0000","last_fullsized":"2024-04-22T01:55:28.388358+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-22T01:55:28.275472+0000","last_change":"2024-04-22T01:55:25.303769+0000","last_active":"2024-04-22T01:55:28.275472+0000","last_peered":"2024-04-22T01:55:28.275472+0000","last_clean":"2024-04-22T01:55:28.275472+0000","last_became_active":"2024-04-22T01:55:25.298462+0000","last_became_peered":"2024-04-22T01:55:25.298462+0000","last_unstale":"2024-04-22T01:55:28.275472+0000","last_undegraded":"2024-04-22T01:55:28.275472+0000","last_fullsized":"2024-04-22T01:55:28.275472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":29768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":180224,"data_stored":620136,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:55:33.235 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T01:55:33.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-22T01:55:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.340+0000 7f27d6aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.347+0000 7f27d6aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.535 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 15 2024-04-22T01:55:33.535 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 62d4ad10-8dc5-4417-9af4-a9625064a333 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-04-22T01:55:16.002883+0000 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-04-22T01:55:29.261878+0000 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 3 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client jewel 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release squid 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:allow_crimson=true 2024-04-22T01:55:33.536 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 4 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6801/464935848 v2:172.21.15.107:6802/464935848 exists,up 08b6bf71-e816-48c5-b073-b7ec9b5acd4e 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6805/3294686425 v2:172.21.15.107:6804/3294686425 exists,up dbd2b417-bbcf-440d-ad42-3232cbb2b5a8 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6803/3900725868 v2:172.21.15.150:6802/3900725868 exists,up 0fc13a13-783d-4b86-870d-3d83caa6e82d 2024-04-22T01:55:33.537 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6804/3017050861 v2:172.21.15.150:6805/3017050861 exists,up 2c6e979e-5a68-4bff-8304-ef2133dfaf2a 2024-04-22T01:55:33.548 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6801/464935848 v2:172.21.15.107:6802/464935848 exists,up 08b6bf71-e816-48c5-b073-b7ec9b5acd4e', 'osd.1 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.107:6805/3294686425 v2:172.21.15.107:6804/3294686425 exists,up dbd2b417-bbcf-440d-ad42-3232cbb2b5a8', 'osd.2 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6803/3900725868 v2:172.21.15.150:6802/3900725868 exists,up 0fc13a13-783d-4b86-870d-3d83caa6e82d', 'osd.3 up in weight 1 up_from 9 up_thru 13 down_at 0 last_clean_interval [0,0) v2:172.21.15.150:6804/3017050861 v2:172.21.15.150:6805/3017050861 exists,up 2c6e979e-5a68-4bff-8304-ef2133dfaf2a'] 2024-04-22T01:55:33.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mon.a", 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster": "ceph", 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket_mode": "", 2024-04-22T01:55:33.652 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_client_required": "cephx, none", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_cluster_required": "cephx", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_debug": "true", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_required": "cephx", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_supported": "cephx", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio": "true", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_poll_ms": "250", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_reap_max": "16", 2024-04-22T01:55:33.653 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_async_discard_threads": "0", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_block_size": "4096", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_inflight_ios": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_enable_discard": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry": "3", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash": "0", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_hipri": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-22T01:55:33.654 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_type": "", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_alloc_size": "1048576", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_allocator": "hybrid", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_buffered_io": "true", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_for_zeros": "false", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_compact_log_sync": "false", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-22T01:55:33.655 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_log_runway": "4194304", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_prefetch": "1048576", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_flush_size": "524288", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_log_runway": "1048576", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery": "false", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_sync_write": "false", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocation_from_file": "true", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocator": "hybrid", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-22T01:55:33.656 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_create": "true", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_create": "false", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_path": "", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_size": "0", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_path": "", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_preallocate_file": "false", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_size": "107374182400", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_create": "false", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_path": "", 2024-04-22T01:55:33.657 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_size": "100663296", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs": "true", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune": "true", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-22T01:55:33.658 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size": "0", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_type": "2q", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_clone_cow": "true", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-22T01:55:33.659 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_mode": "none", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_csum_type": "crc32c", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_freelist": "false", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-22T01:55:33.660 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-22T01:55:33.661 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_small_allocations": "0", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_read": "true", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_write": "false", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-22T01:55:33.662 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fail_eio": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount": "false", 2024-04-22T01:55:33.663 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_ignore_data_csum": "false", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_op_age": "5.000000", 2024-04-22T01:55:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_alloc_size": "0", 2024-04-22T01:55:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size": "0", 2024-04-22T01:55:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-22T01:55:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-22T01:55:33.665 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_deferred_txc": "32", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size": "0", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_nid_prealloc": "1024", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-22T01:55:33.666 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_retry_disk_reads": "3", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cf": "true", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_mem": "512", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-22T01:55:33.667 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_tracing": "false", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-22T01:55:33.668 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_zero_block_detection": "false", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-22T01:55:33.669 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_signatures": "false", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_version": "2", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_signatures": "false", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_version": "2", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_signatures": "false", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_version": "2", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_sign_messages": "true", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "chdir": "", 2024-04-22T01:55:33.670 INFO:teuthology.orchestra.run.smithi107.stdout: "client_acl_type": "", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_asio_thread_count": "2", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_mid": "0.750000", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_size": "16384", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_caps_release_delay": "5", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_check_pool_perm": "true", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_force_sync_read": "false", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_getattr_caps": "false", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_inject_tick_delay": "0", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_remount": "false", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_dirsize_rbytes": "true", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_force_lazyio": "false", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_fs": "", 2024-04-22T01:55:33.671 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_release_failure": "false", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_inline_size": "4096", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mds_namespace": "", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_metadata": "", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_gid": "-1", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_timeout": "300", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_uid": "-1", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mountpoint": "/", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_notify_timeout": "10", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc": "true", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty": "104857600", 2024-04-22T01:55:33.672 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_objects": "1000", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_size": "209715200", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_target_dirty": "8388608", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_permissions": "true", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota": "true", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota_df": "true", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_bytes": "0", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_periods": "4", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_min": "131072", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_reconnect_stale": "false", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_shutdown_timeout": "30", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_snapdir": ".snap", 2024-04-22T01:55:33.673 INFO:teuthology.orchestra.run.smithi107.stdout: "client_tick_interval": "1", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_trace": "", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_try_dentry_invalidate": "false", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_faked_inos": "false", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_random_mds": "false", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog": "false", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_port": "12201", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_monitors": "default=true", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog": "false", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_level": "info", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "-", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network": "", 2024-04-22T01:55:33.674 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network_interface": "", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_isal": "false", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_level": "5", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_winsize": "-15", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zstd_level": "1", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_alien_op_num_threads": "6", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_stat_interval": "0", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_seastar_cpu_cores": "", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location": "", 2024-04-22T01:55:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook": "", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook_timeout": "10", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "daemonize": "false", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "daos_pool": "tank", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_alienstore": "0/5", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_allow_any_pool_priority": "false", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok": "1/5", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok_assert_abort": "false", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asserts_on_shutdown": "true", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_auth": "20/20", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bdev": "1/3", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluefs": "1/5", 2024-04-22T01:55:33.676 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluestore": "1/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_buffer": "0/1", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ceph_exporter": "1/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephfs_mirror": "0/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephsqlite": "0/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_client": "0/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_compressor": "1/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_context": "0/1", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crimson_interrupt": "0/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crush": "1/1", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crypto": "1/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cyanstore": "0/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_deliberately_leak_memory": "false", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_disable_randomized_ping": "false", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_dpdk": "1/5", 2024-04-22T01:55:33.677 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_eventtrace": "1/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filer": "0/1", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filestore": "1/3", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_finisher": "1/1", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_fuse": "1/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeat_testing_span": "0", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeatmap": "1/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_javaclient": "1/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journal": "1/3", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journaler": "0/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_kstore": "1/5", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_lockdep": "0/1", 2024-04-22T01:55:33.678 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mclock": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_balancer": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_locker": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log_expire": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_migrator": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_quiesce": "3/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_memstore": "1/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgr": "2/5", 2024-04-22T01:55:33.679 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgrc": "1/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mon": "20/20", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_monc": "0/10", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ms": "1/1", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_none": "0/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objclass": "0/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objectcacher": "0/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objecter": "0/1", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_optracker": "0/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_osd": "1/5", 2024-04-22T01:55:33.680 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_paxos": "20/20", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_perfcounter": "1/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_prioritycache": "1/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rados": "0/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd": "0/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_mirror": "0/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_pwl": "0/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_replay": "0/5", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_refs": "0/0", 2024-04-22T01:55:33.681 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_reserver": "1/1", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_access": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_datacache": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_dbstore": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_flight": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_notification": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_sync": "1/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rocksdb": "4/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore": "0/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_backref": "0/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_cache": "0/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_cleaner": "0/5", 2024-04-22T01:55:33.682 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_device": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_epm": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_journal": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_lba": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_odata": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_omap": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_onode": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_t": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_tm": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_striper": "0/1", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_test": "0/5", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_throttle": "1/1", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_timer": "0/1", 2024-04-22T01:55:33.683 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_tp": "0/5", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_trace": "1/5", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "device_failure_prediction_mode": "none", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_graylog": "false", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_journald": "false", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_stderr": "true", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_syslog": "false", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event_tracing": "false", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_addr": "0.0.0.0", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_http_port": "9926", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_prio_limit": "5", 2024-04-22T01:55:33.684 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_sort_metrics": "true", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_stats_period": "5", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "fake_statfs_for_testing": "0", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "fatal_signal_handlers": "true", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_purge_ops": "10", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_truncate_ops": "128", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_apply_finisher_threads": "1", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_blackhole": "false", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_clone_range": "true", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_snap": "true", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_caller_concurrency": "10", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_collect_device_partition_information": "true", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_commit_timeout": "600.000000", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_inject_read_err": "false", 2024-04-22T01:55:33.685 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_omap_check": "false", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_verify_split": "false", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_dump_file": "", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fadvise": "true", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fail_eio": "true", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_shards": "16", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_size": "128", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap": "false", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap_threshold": "4096", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_inject_stall": "0", 2024-04-22T01:55:33.686 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_parallel": "false", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_trailing": "false", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_writeahead": "false", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_kill_at": "0", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs": "0", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-22T01:55:33.687 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size": "0", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_merge_threshold": "-10", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_odsync_write": "false", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend": "rocksdb", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend_path": "", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-22T01:55:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_timeout": "60", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_threads": "2", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_punch_hole": "false", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_ops": "50", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-22T01:55:33.689 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_seek_data_hole": "false", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc": "false", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_splice": "false", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_multiple": "2", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_rand_factor": "20", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_update_to": "1000", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-22T01:55:33.690 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_enable": "true", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_xfs_extsize": "false", 2024-04-22T01:55:33.691 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_zfs_snap": "false", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fio_dir": "/tmp/fio", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "62d4ad10-8dc5-4417-9af4-a9625064a333", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_allow_other": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_atomic_o_trunc": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_big_writes": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_debug": "false", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_default_permissions": "false", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_disable_pagecache": "false", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_max_write": "0", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_multithreaded": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_require_active_mds": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_set_user_groups": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_move": "true", 2024-04-22T01:55:33.692 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_read": "true", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_write": "true", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_use_invalidate_cb": "true", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_target_name": "ceph", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_file": "", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_inject_failure": "0", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_interval": "5", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "host": "smithi107", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-22T01:55:33.693 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-22T01:55:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: "inject_early_sigterm": "false", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "jaeger_agent_port": "6799", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "jaeger_tracing_enable": "false", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_aio": "true", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_align_min_size": "65536", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_align": "true", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_size": "4096", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_dio": "true", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_discard": "false", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_force_aio": "false", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_ignore_corruption": "false", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_bytes": "10485760", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_entries": "100", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_replay_from": "0", 2024-04-22T01:55:33.695 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_write_header_frequency": "0", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_zero_on_create": "false", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prefetch_periods": "10", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prezero_periods": "5", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_write_head_interval": "15", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "keyfile": "", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_backend": "rocksdb", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_default_stripe_size": "65536", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount": "false", 2024-04-22T01:55:33.696 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_bytes": "67108864", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_ops": "512", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_nid_prealloc": "1024", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_onode_map_size": "1024", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_submit_transaction": "false", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_transaction": "false", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "librados_thread_count": "2", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep": "false", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep_force_backtrace": "false", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "log_coarse_timestamps": "true", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "log_flush_on_exit": "false", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_host": "127.0.0.1", 2024-04-22T01:55:33.697 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_port": "12201", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_new": "1000", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_recent": "10000", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stderr_prefix": "", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stop_at_utilization": "0.970000", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": "true", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_graylog": "false", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_journald": "false", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_stderr": "false", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_syslog": "false", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "max_rotating_auth_attempts": "10", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_action_on_write_error": "1", 2024-04-22T01:55:33.698 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_alternate_name_max": "8192", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_asio_thread_count": "2", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_export_pin": "true", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_dirs": "true", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_interval": "5", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_interval": "10", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max": "-1", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max_until": "-1", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_merge_size": "50", 2024-04-22T01:55:33.699 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_midchunk": "0.300000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_start": "0.200000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_minchunk": "0.001000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_mode": "0", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_max": "1.200000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_min": "0.800000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_overload_epochs": "2", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_bits": "3", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_size": "10000", 2024-04-22T01:55:33.700 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_target_decay": "10.000000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_grace": "15.000000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_interval": "4.000000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_mid": "0.700000", 2024-04-22T01:55:33.701 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_delay": "0", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_release_free_interval": "10", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_reservation": "0.050000", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_interval": "1", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_threshold": "262144", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-22T01:55:33.702 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_prealloc_inos": "1000", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_connect_bootstrapping": "false", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_damage_table_max_entries": "10000", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_auth_pins": "false", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_frag": "false", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_scatterstat": "false", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_subtrees": "false", 2024-04-22T01:55:33.703 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_decay_halflife": "5.000000", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_default_dir_hash": "2", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_defer_session_stale": "true", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_deny_all_reconnect": "false", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_keys_per_op": "16384", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_commit_size": "10", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_entries": "0", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_prefetch": "true", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_on_map": "false", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_early_reply": "true", 2024-04-22T01:55:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enable_op_tracker": "true", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enforce_unique_name": "true", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random": "true", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_summarize_threshold": "10", 2024-04-22T01:55:33.705 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_health_dummy": "false", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_migrator_session_race": "false", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_join_fs": "", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_journal_format": "1", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_create_at": "0", 2024-04-22T01:55:33.706 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_export_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_import_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_expire_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_replay_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_link_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_mdstable_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_openc_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_rename_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_shutdown_at": "0", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_event_large_threshold": "524288", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_events_per_segment": "1024", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_events": "-1", 2024-04-22T01:55:33.707 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_segments": "128", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_pause": "false", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_segment_size": "0", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_threshold": "128", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_warn_factor": "2.000000", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_caps_per_client": "1048576", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_flushes": "100000", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_requests": "100000", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_export_size": "20971520", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_file_recover": "32", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_files": "64", 2024-04-22T01:55:33.708 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops": "8192", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_snaps_per_dir": "100", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_metrics_update_interval": "2", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_per_client": "100", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_working_set": "10000", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_numa_node": "-1", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_complaint_time": "30.000000", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_duration": "600", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_size": "20", 2024-04-22T01:55:33.709 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_slow_op_size": "20", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_log_threshold": "5", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_grace": "15", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_interval": "5", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_caps": "30000", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_threshold": "262144", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-22T01:55:33.710 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_interval": "1.000000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_gid": "0", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_uid": "0", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-22T01:55:33.711 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_shutdown_check": "0", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_skip_ino": "0", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_max_uid": "4294967294", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_min_uid": "0", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_rstat": "false", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_standby_replay_damaged": "false", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_symlink_recovery": "true", 2024-04-22T01:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_exports": "0", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_fragments": "0", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_tick_interval": "5.000000", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_valgrind_exit": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_backtrace": "1", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_scatter": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_sessions": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool_debug": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_device_bytes": "1073741824", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_set": "false", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_size": "65536", 2024-04-22T01:55:33.713 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_bytes": "134217728", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_messages": "512", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_disabled_modules": "", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_max_pg_creating": "1024", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_max_pg_num_change": "128", 2024-04-22T01:55:33.714 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_bytes": "134217728", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_messages": "128", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_bytes": "134217728", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_messages": "128", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_bytes": "536870912", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_messages": "8192", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_pool": "true", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_standby_modules": "true", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_period": "5", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_threshold": "5", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_tick_period": "2", 2024-04-22T01:55:33.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_delete": "true", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_size_one": "true", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_auth_validate_all_caps": "true", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_bytes": "104857600", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_directed_command_retry": "2", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_parallel": "3", 2024-04-22T01:55:33.716 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_log_interval": "1.000000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_interval": "10.000000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_target_rank": "-1", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_level": "debug", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_file": "true", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-22T01:55:33.717 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_journald": "false", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_bootstrap": "false", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_start": "false", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_trim": "true", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cpu_threads": "4", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_crush_min_required_version": "hammer", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_daemon_bytes": "419430400", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-22T01:55:33.718 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_crit": "5", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_warn": "5", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_size_warn": "16106127360", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_json": "false", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-22T01:55:33.719 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_transactions": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_extra_checks": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_reef": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_squid": "false", 2024-04-22T01:55:33.720 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_down_mkfs_grace": "120", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_down_uptime_grace": "60", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_default_strategy": "1", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_timeout": "5.000000", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_divisor": "2", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_enable_op_tracker": "true", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_fake_pool_delete": "false", 2024-04-22T01:55:33.721 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_force_quorum_join": "false", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_globalid_prealloc": "10000", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_detail_to_clog": "false", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_log_update_period": "5", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_max_detail": "50", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog": "true", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_interval": "600", 2024-04-22T01:55:33.722 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host": "172.21.15.107", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host_override": "", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_initial_members": "", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease": "5.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-22T01:55:33.723 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_full_interval": "50", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max": "10000", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max_summary": "50", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_epochs": "500", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_osd": "10000", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pg_per_osd": "10000", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pool_pg_num": "65536", 2024-04-22T01:55:33.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_force_trim_to": "0", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_skip_sanity": "false", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_autotune": "true", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_target": "2147483648", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_beacon_grace": "30", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_digest_period": "5", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_inactive_grace": "60", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_min_osdmap_epochs": "50", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_complaint_time": "30", 2024-04-22T01:55:33.725 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_duration": "600", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_size": "20", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_size": "20", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_log_threshold": "5", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_in": "false", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size": "500", 2024-04-22T01:55:33.726 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_interval": "600", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_force_trim_to": "0", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_down_reporters": "2", 2024-04-22T01:55:33.727 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_report_timeout": "900", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-22T01:55:33.728 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_stuck_threshold": "60", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_probe_timeout": "2.000000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_change": "0.050000", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_osds": "4", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-22T01:55:33.729 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_interval": "86400", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_max_keys": "100", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_timeout": "300", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_session_timeout": "300", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_smart_report_timeout": "5", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stat_smooth_intervals": "6", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_min_size": "2", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_size": "4", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-22T01:55:33.730 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_debug": "false", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_provider_kill_at": "0", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_requester_kill_at": "0", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_timeout": "60.000000", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_target_pg_per_osd": "100", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_tick_interval": "5", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_interval": "300.000000", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_use_min_delay_socket": "false", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_older_version_delay": "604800", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-22T01:55:33.731 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_misplaced": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_older_version": "true", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-22T01:55:33.732 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "monmap": "", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_tracing_enabled": "false", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_op_threads": "3", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_cm": "false", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_device_name": "", 2024-04-22T01:55:33.733 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_dscp": "96", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_local_gid": "", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_port_num": "1", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_sl": "3", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_support_srq": "true", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_type": "ib", 2024-04-22T01:55:33.734 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_reap_threshold": "5", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_before_connect": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv4": "true", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv6": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr1": "true", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr2": "true", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_max": "7568", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_min": "6800", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_count": "3", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_delay": "5", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_client": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mds": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mgr": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mon": "false", 2024-04-22T01:55:33.735 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_osd": "false", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_client_mode": "crc", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_mode": "crc", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_type": "", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_compress_secure": "false", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_idle_timeout": "900", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_ready_timeout": "10", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_data": "true", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_header": "true", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bad_msg": "false", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bug": "true", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_old_message": "true", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_skipped_message": "false", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-22T01:55:33.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_coremask": "0xF", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_enable_tso": "true", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hugepages": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_lro": "true", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_memory_channel": "4", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_pmd": "", 2024-04-22T01:55:33.737 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_port_id": "0", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_on_send": "false", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_initial_backoff": "0.200000", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_max": "1.000000", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_type": "", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_network_congestion": "0", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_socket_failures": "0", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_learn_addr_from_peer": "true", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_accept_failures": "4", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_backoff": "15.000000", 2024-04-22T01:55:33.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_client_mode": "crc", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_cluster_mode": "crc", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_service_mode": "crc", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compress_min_size": "1024", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compress_mode": "none", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_min_cost": "65536", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_public_type": "", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_service_mode": "crc", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_listen_backlog": "512", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_nodelay": "true", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_rcvbuf": "0", 2024-04-22T01:55:33.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_type": "async+posix", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "no_config_file": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_completion_locks_per_session": "32", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_ops": "1024", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_tick_interval": "5.000000", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_timeout": "10.000000", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_blackhole": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "openssl_engine_opts": "", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_delay_time": "5.000000", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_hist_halflife": "1000", 2024-04-22T01:55:33.740 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_low_ops": "2", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_ops": "4", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_slop": "0.020000", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_asio_thread_count": "2", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_async_recovery_min_cost": "100", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_max": "512", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_min": "64", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_degraded": "false", 2024-04-22T01:55:33.741 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_peering": "false", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_unfound": "true", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_beacon_report_interval": "300", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_duration": "30", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_max_block_size": "67108864", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blkin_trace_all": "false", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_for_log_corruption": "false", 2024-04-22T01:55:33.742 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_update_on_start": "true", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_cap": "256", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_size_cap": "524288000", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_op_priority": "63", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_watch_timeout": "30", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_max_records": "256", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_compact_on_start": "false", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_on_start": "true", 2024-04-22T01:55:33.743 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_weight_set": "true", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_feed_pullee": "-1", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_misdirected_ops": "false", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_acting_change": "false", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_purge_strays": "false", 2024-04-22T01:55:33.744 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_op_order": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_shutdown": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_trim_objects": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_snaps": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-22T01:55:33.745 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_keys": "1024", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_stride": "524288", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_notify_timeout": "30", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep": "0.000000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_discard_disconnected_ops": "true", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_enable_op_tracker": "true", 2024-04-22T01:55:33.746 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_info": "true", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown": "true", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_function_tracing": "false", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_grace": "20", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_interval": "6", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-22T01:55:33.747 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_peers": "10", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_size": "2000", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_stale": "600", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_max_size": "100000", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_min_size": "1000", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_size": "5120", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_kill_backfill_at": "0", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-22T01:55:33.748 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_cache_size": "50", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_dedup": "true", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max": "40", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_share_max_epochs": "40", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_name_len": "100", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_size": "0", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_backfills": "1", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_count": "5", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_period": "600", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_name_len": "2048", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_namespace_len": "256", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_size": "134217728", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-22T01:55:33.749 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_blocked_by": "16", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_log_entries": "10000", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pgls": "1024", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_cost": "8388608", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_objects": "10", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_scrubs": "3", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_trimming_pgs": "2", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_op_reply_len": "64", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_size": "90", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-22T01:55:33.750 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-22T01:55:33.751 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_profile": "balanced", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-22T01:55:33.753 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_base": "805306368", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_min": "134217728", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target": "4294967296", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_autotune": "false", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_pg_log_entries": "250", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_recovery_priority": "0", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-22T01:55:33.754 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_interval": "5", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_cache_shards": "32", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_op_tracker_shard": "32", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_auto_affinity": "true", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_node": "-1", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_prefer_iface": "true", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objecter_finishers": "1", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore": "bluestore", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_fuse": "false", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_tracing": "false", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_complaint_time": "30.000000", 2024-04-22T01:55:33.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_duration": "600", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_size": "20", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_size": "20", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_log_threshold": "5", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards": "0", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_hdd": "5", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_ssd": "8", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_min_cost": "65536", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-22T01:55:33.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue_cut_off": "high", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_timeout": "15", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_open_classes_on_start": "true", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_os_flags": "0", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_peering_op_priority": "255", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_cost": "1048576", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_priority": "5", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_max": "10000", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_min": "100", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-22T01:55:33.757 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crimson": "true", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-22T01:55:33.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flags": "0", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_min_size": "2", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_num": "32", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pgp_num": "0", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_ratio": "70", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_size": "3", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_type": "replicated", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-22T01:55:33.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_push_per_object_cost": "1000", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap": "true", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_cost": "20971520", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active": "0", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-22T01:55:33.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_single_start": "1", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_priority": "3", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_priority": "5", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep": "0.000000", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_repair_during_recovery": "false", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_requested_scrub_priority": "120", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-22T01:55:33.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair": "false", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_hour": "0", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_week_day": "0", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_max": "25", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_min": "5", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_cost": "52428800", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_during_recovery": "false", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_hour": "0", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_week_day": "0", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_event_cost": "4096", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-22T01:55:33.762 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_invalid_stats": "true", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_preemptions": "5", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_priority": "5", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_sleep": "0.000000", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_data_digest": "false", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_smart_report_timeout": "5", 2024-04-22T01:55:33.763 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_cost": "1048576", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_priority": "5", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_transaction_size": "30", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-22T01:55:33.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tracing": "false", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_use_stale_snap": "false", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osdc_blkin_trace_all": "false", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_kill_at": "0", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_max_join_drift": "10", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min": "500", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min_wait": "0.050000", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_propose_interval": "1.000000", 2024-04-22T01:55:33.765 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max": "500", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_min": "10", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_stash_full_interval": "25", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_max": "500", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_min": "250", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "perf": "true", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "-", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrv": "", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_bind_addr": "-", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network": "", 2024-04-22T01:55:33.766 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network_interface": "", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_enabled": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_session_max_number": "256", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_mon_op_timeout": "0", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_osd_op_timeout": "0", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_tracing": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_atime_update_interval": "60", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_parent_reads": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_snap_reads": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blkin_trace_all": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache": "true", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-22T01:55:33.767 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_policy": "writearound", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_size": "33554432", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_clone_copy_on_read": "false", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_compression_hint": "none", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_concurrent_management_ops": "10", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_clone_format": "auto", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_data_pool": "", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_features": "61", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_format": "2", 2024-04-22T01:55:33.768 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_map_options": "", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_order": "22", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_pool": "rbd", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_count": "0", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_unit": "0", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_enable_alloc_hint": "true", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler": "simple", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-22T01:55:33.769 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_order": "24", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_pool": "", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_splay_width": "4", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_parent_reads": "false", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_snap_reads": "false", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-22T01:55:33.770 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-22T01:55:33.771 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mtime_update_interval": "60", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_non_blocking_aio": "true", 2024-04-22T01:55:33.772 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_thread_timeout": "60", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_threads": "1", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_parent_cache_enabled": "false", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_plugins": "", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst": "0", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_limit": "0", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_exclude_ops": "0", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst": "0", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-22T01:55:33.773 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_limit": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-22T01:55:33.774 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_read_from_replica_policy": "default", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_skip_partial_discard": "true", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_tracing": "false", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_names": "true", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_pool": "true", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_base_url": "", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_log_level": "", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_acl_grants_max_num": "100", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_admin_entry": "admin", 2024-04-22T01:55:33.775 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_backend_store": "rados", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_barbican_url": "", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_beast_enable_async": "true", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_counters_cache": "false", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_enabled": "true", 2024-04-22T01:55:33.776 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_expiry_interval": "900", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_lru_size": "10000", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_config_store": "rados", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_content_length_compat": "false", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress": "true", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cors_rules_max_num": "100", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cross_domain_policy": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_addr": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_password": "", 2024-04-22T01:55:33.777 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_username": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_require_ssl": "true", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-22T01:55:33.778 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_addr": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_auth": "token", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_namespace": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_prefix": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_token_file": "", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-22T01:55:33.779 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_buffersize": "524288", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_time": "300", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-22T01:55:33.780 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_port": "6379", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_changes_size": "1000", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_num_shards": "128", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_window": "30", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-22T01:55:33.781 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-22T01:55:33.782 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_name": "", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_s3website_name": "", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding": "true", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_gc_threads": "true", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_lc_threads": "true", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_ops_log": "false", 2024-04-22T01:55:33.783 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_quota_threads": "true", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_static_website": "false", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_usage_log": "false", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enforce_swift_acls": "true", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_exit_timeout_secs": "120", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_expose_bucket": "false", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_extended_http_attrs": "", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_filter": "none", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontends": "beast port=7480", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred": "50", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_objs": "32", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-22T01:55:33.784 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_period": "3600", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_init_timeout": "300", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_connection_idle": "300", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-22T01:55:33.785 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_domain": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password_path": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_project": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_tenant": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token_path": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_user": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_api_version": "2", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_domain": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_password": "", 2024-04-22T01:55:33.786 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_project": "", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_user": "", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_url": "", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_debug_interval": "-1", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_lock_max_time": "90", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_objs": "32", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_rules": "1000", 2024-04-22T01:55:33.787 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_worker": "3", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_thread_delay": "0", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_dnattr": "uid", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchfilter": "", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_uri": "", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_http_headers": "", 2024-04-22T01:55:33.788 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name_utc": "false", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_name_len": "0", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_size": "0", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_chunk_size": "4194304", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_listing_results": "1000", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_notify_retries": "10", 2024-04-22T01:55:33.789 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_param_size": "1048576", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_size": "5368709120", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_slo_entries": "1000", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_log_max_shards": "64", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mp_lock_max_time": "600", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-22T01:55:33.790 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_max_gc": "300", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_async_rados_threads": "32", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_control_oids": "8", 2024-04-22T01:55:33.791 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_numa_node": "-1", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_chunk_size": "100", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_gc_interval": "600", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_timeout": "600", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_tracing": "false", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_token": "", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_url": "", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_verify_ssl": "true", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-22T01:55:33.792 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_rados": "false", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_socket_path": "", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval": "2.000000", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_lanes": "3", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-22T01:55:33.793 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_partitions": "3", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_continue": "true", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_tracing": "false", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm": "", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_id": "", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region": "", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-22T01:55:33.794 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_request_uri": "", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_batch_size": "64", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_max_aio": "128", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_num_logs": "16", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_thread_interval": "600", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_resolve_cname": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_run_sync_thread": "true", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-22T01:55:33.795 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_scheduler_type": "throttler", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_script_uri": "", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_service_provider_name": "", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_id": "", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_secret": "", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_entry": "sts", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_key": "sts", 2024-04-22T01:55:33.796 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_min_session_duration": "900", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_token_introspection_url": "", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_account_in_url": "false", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_entry": "auth", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_url": "", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_custom_header": "", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_need_stats": "true", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_tenant_name": "", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_token_expiration": "86400", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url": "", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url_prefix": "swift", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-22T01:55:33.797 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_lease_period": "120", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_thread_pool_size": "512", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-22T01:55:33.798 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_comment": "", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_createby": "", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_encoding": "", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_flag": "false", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_origin": "", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_tracker": "", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_trust_forwarded_https": "false", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_shards": "32", 2024-04-22T01:55:33.799 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_user_shards": "1", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_use_opa_authz": "false", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_counters_cache": "false", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_max_buckets": "1000", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_policies_max_num": "100", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_unique_email": "true", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_verify_ssl": "true", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-22T01:55:33.800 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone": "", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_id": "", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup": "", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_id": "", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgwlc_auto_session_clear": "true", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_block_size": "4096", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_size": "536870912", 2024-04-22T01:55:33.801 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_index_type": "binary_search", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_partition_filters": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_perf": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-22T01:55:33.802 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "run_dir": "/var/run/ceph", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_block_create": "true", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_cache_lru_size": "67108864", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_default_max_object_size": "16777216", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_device_size": "53687091200", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_capacity": "16", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_main_device_type": "SSD", 2024-04-22T01:55:33.803 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_segment_size": "67108864", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "set_keepcaps": "false", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "setgroup": "", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser": "", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser_match_path": "", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "thp": "false", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_default_timeout": "60", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout: "throttler_perf_counter": "true" 2024-04-22T01:55:33.804 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:55:33.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-22T01:55:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.911+0000 7fcd9262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:33.916+0000 7fcd9262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:33.929 INFO:teuthology.orchestra.run.smithi107.stdout:{} 2024-04-22T01:55:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-22T01:55:33.940 DEBUG:teuthology.orchestra.run.smithi107:> type ceph-objectstore-tool 2024-04-22T01:55:33.954 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T01:55:33.955 DEBUG:teuthology.orchestra.run.smithi150:> type ceph-objectstore-tool 2024-04-22T01:55:33.972 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-22T01:55:33.973 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-22T01:55:33.983 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-22T01:55:33.984 INFO:tasks.radosbench:Beginning radosbench... 2024-04-22T01:55:33.984 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-22T01:55:33.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-22T01:55:33.985 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-22T01:55:33.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-04-22T01:55:33.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:55:33.985 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2024-04-22T01:55:33.986 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T01:55:33.986 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:55:33.986 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-22T01:55:33.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:55:34.066 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T01:55:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:34.102+0000 7fb8c666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:34.107+0000 7fb8c666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:34.102+0000 7f1789ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:34.114+0000 7f1789ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_0' created 2024-04-22T01:55:35.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-22T01:55:35.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:35.443+0000 7f109715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:35.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:35.456+0000 7f109715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:36.298 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-22T01:55:37.318 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-22T01:55:37.321 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-22T01:55:37.321 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-22T01:55:37.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-04-22T01:55:37.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-22T01:55:37.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_29052 2024-04-22T01:55:37.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:55:37.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-04-22T01:55:38.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 16 0 0 0 - 0 2024-04-22T01:55:39.067 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2024-04-22T01:55:39.067 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:55:39.067 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-22T01:55:39.068 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T01:55:39.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T01:55:39.070 INFO:tasks.ceph.osd.2:Started 2024-04-22T01:55:39.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T01:55:39.180 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:55:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:55:39.184 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T01:55:39.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 16 0 0 0 - 0 2024-04-22T01:55:40.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 16 0 0 0 - 0 2024-04-22T01:55:41.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 16 0 0 0 - 0 2024-04-22T01:55:42.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 16 0 0 0 - 0 2024-04-22T01:55:43.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 16 0 0 0 - 0 2024-04-22T01:55:44.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T01:55:44.294 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-22T01:55:44.294 INFO:teuthology.orchestra.run.smithi150.stdout: "ops_in_flight": [] 2024-04-22T01:55:44.295 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-22T01:55:44.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 16 0 0 0 - 0 2024-04-22T01:55:45.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 16 0 0 0 - 0 2024-04-22T01:55:46.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:55:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:46.286+0000 7f93644ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:46.292+0000 7f93644ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:46.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 16 0 0 0 - 0 2024-04-22T01:55:47.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 16 0 0 0 - 0 2024-04-22T01:55:48.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:55:48.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 16 0 0 0 - 0 2024-04-22T01:55:48.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:48.500+0000 7f2a16224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:48.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:48.505+0000 7f2a16224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:49.305 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 2] 2024-04-22T01:55:49.306 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:55:49.306 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 3, 2] 2024-04-22T01:55:49.306 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T01:55:49.307 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:55:49.362 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T01:55:49.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 16 0 0 0 - 0 2024-04-22T01:55:50.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 16 0 0 0 - 0 2024-04-22T01:55:50.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:55:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:50.718+0000 7fdf648ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:50.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:50.723+0000 7fdf648ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:51.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 16 0 0 0 - 0 2024-04-22T01:55:52.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 16 0 0 0 - 0 2024-04-22T01:55:53.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 40 24 0.0117174 0.0117188 0.0711173 10.1242 2024-04-22T01:55:54.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 3, 2] 2024-04-22T01:55:54.363 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:55:54.363 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-22T01:55:54.363 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T01:55:54.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T01:55:54.366 INFO:tasks.ceph.osd.1:Started 2024-04-22T01:55:54.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:55:54.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 40 24 0.0110281 0 - 10.1242 2024-04-22T01:55:54.471 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:55:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:55:54.475 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T01:55:55.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 40 24 0.0104155 0 - 10.1242 2024-04-22T01:55:56.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 40 24 0.0098673 0 - 10.1242 2024-04-22T01:55:57.338 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:55:57.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:55:57.389950+0000 min lat: 0.0383534 max lat: 15.2689 avg lat: 10.1242 2024-04-22T01:55:57.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:55:57.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 40 24 0.00937395 0 - 10.1242 2024-04-22T01:55:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:55:57.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:55:58.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 40 24 0.00892757 0 - 10.1242 2024-04-22T01:55:59.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 40 24 0.00852178 0 - 10.1242 2024-04-22T01:55:59.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:55:59.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:55:59.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:59.583+0000 7f01f8b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:59.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:55:59.590+0000 7f01f8b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [ 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "peering_event", 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 177, 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "PeeringEvent": { 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "0", 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "3.9", 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "sent": 25, 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "requested": 25, 2024-04-22T01:55:59.673 INFO:teuthology.orchestra.run.smithi107.stdout: "evt": "epoch_sent: 25 epoch_requested: 25 MInfoRec from 0 info: 3.9( v 22'1 (0'0,22'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) pg_lease(ru 55.354961395s ub 55.354961395s int 16.000000000s)", 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.662176+0000" 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.662183+0000" 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.663647+0000" 2024-04-22T01:55:59.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.663837+0000" 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.663986+0000" 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.664103+0000" 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:55:59.664439+0000" 2024-04-22T01:55:59.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "replicated_request", 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 178, 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "RepRequest": { 2024-04-22T01:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: "reqid": "client.4233.0:18", 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "3.9", 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 25, 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "min_epoch": 24, 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "oid": "3:92c12906:::benchmark_data_smithi107_29052_object2:head", 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "0" 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "replicated_request", 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 176, 2024-04-22T01:55:59.678 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "RepRequest": { 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "reqid": "client.4233.0:34", 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "3.0", 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 25, 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "min_epoch": 24, 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "oid": "3:0e445421:::benchmark_data_smithi107_29052_object4:head", 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "3" 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:55:59.680 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:55:59.680 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:56:00.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 96 80 0.0271709 0.0625 0.0467336 4.51212 2024-04-22T01:56:01.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 96 80 0.0260388 0 - 4.51212 2024-04-22T01:56:01.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:01.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:01.846+0000 7fddeee17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:01.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:01.852+0000 7fddeee17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:02.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 96 80 0.0249972 0 - 4.51212 2024-04-22T01:56:03.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 96 80 0.0240357 0 - 4.51212 2024-04-22T01:56:03.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:04.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:04.065+0000 7f4a02f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:04.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:04.071+0000 7f4a02f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:04.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 96 80 0.0231454 0 - 4.51212 2024-04-22T01:56:04.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 1] 2024-04-22T01:56:04.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:04.687 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 3, 2, 1] 2024-04-22T01:56:04.687 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:56:04.687 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:56:04.758 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:56:05.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 96 80 0.0223188 0 - 4.51212 2024-04-22T01:56:06.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:06.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:06.282+0000 7f677742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:06.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:06.288+0000 7f677742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:06.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 96 80 0.0215492 0 - 4.51212 2024-04-22T01:56:07.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 96 80 0.0208308 0 - 4.51212 2024-04-22T01:56:08.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 96 80 0.0201588 0 - 4.51212 2024-04-22T01:56:09.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 96 80 0.0195289 0 - 4.51212 2024-04-22T01:56:09.759 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [3, 2, 1] 2024-04-22T01:56:09.759 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-22T01:56:09.759 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:56:09.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:56:09.762 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:56:09.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:56:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:56:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:09.875 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:56:10.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 96 80 0.0189371 0 - 4.51212 2024-04-22T01:56:11.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 96 80 0.0183801 0 - 4.51212 2024-04-22T01:56:12.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 96 80 0.017855 0 - 4.51212 2024-04-22T01:56:13.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 96 80 0.017359 0 - 4.51212 2024-04-22T01:56:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:56:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:56:14.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:56:14.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 96 80 0.0168898 0 - 4.51212 2024-04-22T01:56:14.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:56:14.983 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:56:14.983 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [] 2024-04-22T01:56:14.983 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:56:14.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T01:56:15.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:15.124+0000 7f1e8aead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:15.142+0000 7f1e8aead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:15.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 96 80 0.0164454 0 - 4.51212 2024-04-22T01:56:16.026 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T01:56:16.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T01:56:16.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:16.187+0000 7f746b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:16.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:16.205+0000 7f746b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:16.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:16.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:16.374+0000 7f49738ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:16.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:16.382+0000 7f49738ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:16.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 100 84 0.0168249 0.00195312 16.5073 5.08156 2024-04-22T01:56:17.033 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T01:56:17.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T01:56:17.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:17.156+0000 7fa0ee7c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:17.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:17.165+0000 7fa0ee7c6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:17.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:56:17.392517+0000 min lat: 0.019524 max lat: 16.5699 avg lat: 4.36373 2024-04-22T01:56:17.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:56:17.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 160 144 0.0281216 0.46875 0.0672018 4.36373 2024-04-22T01:56:18.034 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T01:56:18.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T01:56:18.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:18.156+0000 7fa9d1a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:18.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:18.166+0000 7fa9d1a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:18.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 160 144 0.0274357 0 - 4.36373 2024-04-22T01:56:18.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:18.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:18.686+0000 7f700b79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:18.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:18.691+0000 7f700b79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.044 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T01:56:19.065 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T01:56:19.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.171+0000 7f01cb86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.178+0000 7f01cb86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:56:19.362 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:56:19.374 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-22T01:56:19.048618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1770016,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":281,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110108,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377377252,"statfs":{"total":386547056640,"available":386434306048,"internally_reserved":0,"allocated":2748416,"data_stored":5359199,"data_compressed":258379,"data_compressed_allocated":1761280,"data_compressed_original":4997120,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":516096,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025252"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.460693+0000","last_change":"2024-04-22T01:55:59.462390+0000","last_active":"2024-04-22T01:55:59.460693+0000","last_peered":"2024-04-22T01:55:59.460693+0000","last_clean":"2024-04-22T01:55:46.223396+0000","last_became_active":"2024-04-22T01:55:59.446312+0000","last_became_peered":"2024-04-22T01:55:59.446312+0000","last_unstale":"2024-04-22T01:56:14.023963+0000","last_undegraded":"2024-04-22T01:55:59.460693+0000","last_fullsized":"2024-04-22T01:55:59.460693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.065226+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:56:16.065226+0000","last_peered":"2024-04-22T01:56:16.065226+0000","last_clean":"2024-04-22T01:56:16.065226+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:56:16.065226+0000","last_undegraded":"2024-04-22T01:56:16.065226+0000","last_fullsized":"2024-04-22T01:56:16.065226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'16","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.067293+0000","last_change":"2024-04-22T01:56:16.167448+0000","last_active":"2024-04-22T01:56:18.067293+0000","last_peered":"2024-04-22T01:56:18.067293+0000","last_clean":"2024-04-22T01:56:18.067293+0000","last_became_active":"2024-04-22T01:56:16.159250+0000","last_became_peered":"2024-04-22T01:56:16.159250+0000","last_unstale":"2024-04-22T01:56:18.067293+0000","last_undegraded":"2024-04-22T01:56:18.067293+0000","last_fullsized":"2024-04-22T01:56:18.067293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.386865+0000","last_change":"2024-04-22T01:55:44.499032+0000","last_active":"2024-04-22T01:55:59.386865+0000","last_peered":"2024-04-22T01:55:59.386865+0000","last_clean":"2024-04-22T01:55:59.386865+0000","last_became_active":"2024-04-22T01:55:44.493349+0000","last_became_peered":"2024-04-22T01:55:44.493349+0000","last_unstale":"2024-04-22T01:56:14.023965+0000","last_undegraded":"2024-04-22T01:55:59.386865+0000","last_fullsized":"2024-04-22T01:55:59.386865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'8","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.061816+0000","last_change":"2024-04-22T01:56:16.179472+0000","last_active":"2024-04-22T01:56:17.061816+0000","last_peered":"2024-04-22T01:56:17.061816+0000","last_clean":"2024-04-22T01:56:17.061816+0000","last_became_active":"2024-04-22T01:56:16.170620+0000","last_became_peered":"2024-04-22T01:56:16.170620+0000","last_unstale":"2024-04-22T01:56:17.061816+0000","last_undegraded":"2024-04-22T01:56:17.061816+0000","last_fullsized":"2024-04-22T01:56:17.061816+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"25'7","reported_seq":34,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.795000+0000","last_change":"2024-04-22T01:55:59.442173+0000","last_active":"2024-04-22T01:55:59.795000+0000","last_peered":"2024-04-22T01:55:59.795000+0000","last_clean":"2024-04-22T01:55:59.795000+0000","last_became_active":"2024-04-22T01:55:59.435623+0000","last_became_peered":"2024-04-22T01:55:59.435623+0000","last_unstale":"2024-04-22T01:56:14.023967+0000","last_undegraded":"2024-04-22T01:55:59.795000+0000","last_fullsized":"2024-04-22T01:55:59.795000+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"25'7","reported_seq":38,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.756631+0000","last_change":"2024-04-22T01:55:59.629702+0000","last_active":"2024-04-22T01:55:59.756631+0000","last_peered":"2024-04-22T01:55:59.756631+0000","last_clean":"2024-04-22T01:55:59.756631+0000","last_became_active":"2024-04-22T01:55:59.453121+0000","last_became_peered":"2024-04-22T01:55:59.453121+0000","last_unstale":"2024-04-22T01:56:14.023969+0000","last_undegraded":"2024-04-22T01:55:59.756631+0000","last_fullsized":"2024-04-22T01:55:59.756631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'16","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.071027+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:56:18.071027+0000","last_peered":"2024-04-22T01:56:18.071027+0000","last_clean":"2024-04-22T01:56:18.071027+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:56:18.071027+0000","last_undegraded":"2024-04-22T01:56:18.071027+0000","last_fullsized":"2024-04-22T01:56:18.071027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":192,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.051297+0000","last_change":"2024-04-22T01:56:16.145535+0000","last_active":"2024-04-22T01:56:17.051297+0000","last_peered":"2024-04-22T01:56:17.051297+0000","last_clean":"2024-04-22T01:56:17.051297+0000","last_became_active":"2024-04-22T01:56:16.138886+0000","last_became_peered":"2024-04-22T01:56:16.138886+0000","last_unstale":"2024-04-22T01:56:17.051297+0000","last_undegraded":"2024-04-22T01:56:17.051297+0000","last_fullsized":"2024-04-22T01:56:17.051297+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.068122+0000","last_change":"2024-04-22T01:56:16.170099+0000","last_active":"2024-04-22T01:56:18.068122+0000","last_peered":"2024-04-22T01:56:18.068122+0000","last_clean":"2024-04-22T01:56:18.068122+0000","last_became_active":"2024-04-22T01:56:16.163854+0000","last_became_peered":"2024-04-22T01:56:16.163854+0000","last_unstale":"2024-04-22T01:56:18.068122+0000","last_undegraded":"2024-04-22T01:56:18.068122+0000","last_fullsized":"2024-04-22T01:56:18.068122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'16","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.059231+0000","last_change":"2024-04-22T01:56:16.277560+0000","last_active":"2024-04-22T01:56:18.059231+0000","last_peered":"2024-04-22T01:56:18.059231+0000","last_clean":"2024-04-22T01:56:18.059231+0000","last_became_active":"2024-04-22T01:56:16.140886+0000","last_became_peered":"2024-04-22T01:56:16.140886+0000","last_unstale":"2024-04-22T01:56:18.059231+0000","last_undegraded":"2024-04-22T01:56:18.059231+0000","last_fullsized":"2024-04-22T01:56:18.059231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":44,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.453870+0000","last_change":"2024-04-22T01:55:59.455186+0000","last_active":"2024-04-22T01:55:59.453870+0000","last_peered":"2024-04-22T01:55:59.453870+0000","last_clean":"2024-04-22T01:55:46.214037+0000","last_became_active":"2024-04-22T01:55:59.449321+0000","last_became_peered":"2024-04-22T01:55:59.449321+0000","last_unstale":"2024-04-22T01:56:14.023972+0000","last_undegraded":"2024-04-22T01:55:59.453870+0000","last_fullsized":"2024-04-22T01:55:59.453870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"25'8","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.057000+0000","last_change":"2024-04-22T01:56:16.132726+0000","last_active":"2024-04-22T01:56:17.057000+0000","last_peered":"2024-04-22T01:56:17.057000+0000","last_clean":"2024-04-22T01:56:17.057000+0000","last_became_active":"2024-04-22T01:56:16.127772+0000","last_became_peered":"2024-04-22T01:56:16.127772+0000","last_unstale":"2024-04-22T01:56:17.057000+0000","last_undegraded":"2024-04-22T01:56:17.057000+0000","last_fullsized":"2024-04-22T01:56:17.057000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.070601+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:56:18.070601+0000","last_peered":"2024-04-22T01:56:18.070601+0000","last_clean":"2024-04-22T01:56:18.070601+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:56:18.070601+0000","last_undegraded":"2024-04-22T01:56:18.070601+0000","last_fullsized":"2024-04-22T01:56:18.070601+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"28'16","reported_seq":58,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.064142+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:56:17.064142+0000","last_peered":"2024-04-22T01:56:17.064142+0000","last_clean":"2024-04-22T01:56:17.064142+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:56:17.064142+0000","last_undegraded":"2024-04-22T01:56:17.064142+0000","last_fullsized":"2024-04-22T01:56:17.064142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.455988+0000","last_change":"2024-04-22T01:55:59.457416+0000","last_active":"2024-04-22T01:55:59.455988+0000","last_peered":"2024-04-22T01:55:59.455988+0000","last_clean":"2024-04-22T01:55:46.220728+0000","last_became_active":"2024-04-22T01:55:59.440651+0000","last_became_peered":"2024-04-22T01:55:59.440651+0000","last_unstale":"2024-04-22T01:56:14.023984+0000","last_undegraded":"2024-04-22T01:55:59.455988+0000","last_fullsized":"2024-04-22T01:55:59.455988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.165174+0000","last_change":"2024-04-22T01:56:16.167034+0000","last_active":"2024-04-22T01:56:16.165174+0000","last_peered":"2024-04-22T01:56:16.165174+0000","last_clean":"2024-04-22T01:55:59.913034+0000","last_became_active":"2024-04-22T01:56:16.159121+0000","last_became_peered":"2024-04-22T01:56:16.159121+0000","last_unstale":"2024-04-22T01:56:16.165174+0000","last_undegraded":"2024-04-22T01:56:16.165174+0000","last_fullsized":"2024-04-22T01:56:16.165174+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"28'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069657+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:56:18.069657+0000","last_peered":"2024-04-22T01:56:18.069657+0000","last_clean":"2024-04-22T01:56:18.069657+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:56:18.069657+0000","last_undegraded":"2024-04-22T01:56:18.069657+0000","last_fullsized":"2024-04-22T01:56:18.069657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.161009+0000","last_change":"2024-04-22T01:56:16.162725+0000","last_active":"2024-04-22T01:56:16.161009+0000","last_peered":"2024-04-22T01:56:16.161009+0000","last_clean":"2024-04-22T01:55:59.377500+0000","last_became_active":"2024-04-22T01:56:16.156228+0000","last_became_peered":"2024-04-22T01:56:16.156228+0000","last_unstale":"2024-04-22T01:56:16.161009+0000","last_undegraded":"2024-04-22T01:56:16.161009+0000","last_fullsized":"2024-04-22T01:56:16.161009+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.147048+0000","last_change":"2024-04-22T01:56:16.148704+0000","last_active":"2024-04-22T01:56:16.147048+0000","last_peered":"2024-04-22T01:56:16.147048+0000","last_clean":"2024-04-22T01:55:59.371075+0000","last_became_active":"2024-04-22T01:56:16.142439+0000","last_became_peered":"2024-04-22T01:56:16.142439+0000","last_unstale":"2024-04-22T01:56:16.147048+0000","last_undegraded":"2024-04-22T01:56:16.147048+0000","last_fullsized":"2024-04-22T01:56:16.147048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.060838+0000","last_change":"2024-04-22T01:56:16.165211+0000","last_active":"2024-04-22T01:56:17.060838+0000","last_peered":"2024-04-22T01:56:17.060838+0000","last_clean":"2024-04-22T01:56:17.060838+0000","last_became_active":"2024-04-22T01:56:16.155541+0000","last_became_peered":"2024-04-22T01:56:16.155541+0000","last_unstale":"2024-04-22T01:56:17.060838+0000","last_undegraded":"2024-04-22T01:56:17.060838+0000","last_fullsized":"2024-04-22T01:56:17.060838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.063441+0000","last_change":"2024-04-22T01:56:16.165084+0000","last_active":"2024-04-22T01:56:17.063441+0000","last_peered":"2024-04-22T01:56:17.063441+0000","last_clean":"2024-04-22T01:56:17.063441+0000","last_became_active":"2024-04-22T01:56:16.158792+0000","last_became_peered":"2024-04-22T01:56:16.158792+0000","last_unstale":"2024-04-22T01:56:17.063441+0000","last_undegraded":"2024-04-22T01:56:17.063441+0000","last_fullsized":"2024-04-22T01:56:17.063441+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.188165+0000","last_change":"2024-04-22T01:56:16.189849+0000","last_active":"2024-04-22T01:56:16.188165+0000","last_peered":"2024-04-22T01:56:16.188165+0000","last_clean":"2024-04-22T01:55:59.386495+0000","last_became_active":"2024-04-22T01:56:16.183739+0000","last_became_peered":"2024-04-22T01:56:16.183739+0000","last_unstale":"2024-04-22T01:56:16.188165+0000","last_undegraded":"2024-04-22T01:56:16.188165+0000","last_fullsized":"2024-04-22T01:56:16.188165+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.061099+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:56:16.061099+0000","last_peered":"2024-04-22T01:56:16.061099+0000","last_clean":"2024-04-22T01:56:16.061099+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:56:16.061099+0000","last_undegraded":"2024-04-22T01:56:16.061099+0000","last_fullsized":"2024-04-22T01:56:16.061099+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069950+0000","last_change":"2024-04-22T01:56:16.171487+0000","last_active":"2024-04-22T01:56:18.069950+0000","last_peered":"2024-04-22T01:56:18.069950+0000","last_clean":"2024-04-22T01:56:18.069950+0000","last_became_active":"2024-04-22T01:56:16.165037+0000","last_became_peered":"2024-04-22T01:56:16.165037+0000","last_unstale":"2024-04-22T01:56:18.069950+0000","last_undegraded":"2024-04-22T01:56:18.069950+0000","last_fullsized":"2024-04-22T01:56:18.069950+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27632,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":827392,"data_stored":1721095,"data_compressed":83889,"data_compressed_allocated":565248,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27592,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":720896,"data_stored":1521242,"data_compressed":73761,"data_compressed_allocated":466944,"data_compressed_original":1425408,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27572,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":700416,"data_stored":999655,"data_compressed":47196,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":499712,"data_stored":1117207,"data_compressed":53533,"data_compressed_allocated":270336,"data_compressed_original":1032192,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:56:19.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:56:19.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 160 144 0.0267825 0 - 4.36373 2024-04-22T01:56:19.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.479+0000 7f122a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.487+0000 7f122a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.671 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:56:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:56:19.684 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-22T01:56:19.048618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1770016,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":281,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110108,"kb_used_data":2684,"kb_used_omap":0,"kb_used_meta":107392,"kb_avail":377377252,"statfs":{"total":386547056640,"available":386434306048,"internally_reserved":0,"allocated":2748416,"data_stored":5359199,"data_compressed":258379,"data_compressed_allocated":1761280,"data_compressed_original":4997120,"omap_allocated":0,"internal_metadata":109969408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":516096,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025252"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.460693+0000","last_change":"2024-04-22T01:55:59.462390+0000","last_active":"2024-04-22T01:55:59.460693+0000","last_peered":"2024-04-22T01:55:59.460693+0000","last_clean":"2024-04-22T01:55:46.223396+0000","last_became_active":"2024-04-22T01:55:59.446312+0000","last_became_peered":"2024-04-22T01:55:59.446312+0000","last_unstale":"2024-04-22T01:56:14.023963+0000","last_undegraded":"2024-04-22T01:55:59.460693+0000","last_fullsized":"2024-04-22T01:55:59.460693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.065226+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:56:16.065226+0000","last_peered":"2024-04-22T01:56:16.065226+0000","last_clean":"2024-04-22T01:56:16.065226+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:56:16.065226+0000","last_undegraded":"2024-04-22T01:56:16.065226+0000","last_fullsized":"2024-04-22T01:56:16.065226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'16","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.067293+0000","last_change":"2024-04-22T01:56:16.167448+0000","last_active":"2024-04-22T01:56:18.067293+0000","last_peered":"2024-04-22T01:56:18.067293+0000","last_clean":"2024-04-22T01:56:18.067293+0000","last_became_active":"2024-04-22T01:56:16.159250+0000","last_became_peered":"2024-04-22T01:56:16.159250+0000","last_unstale":"2024-04-22T01:56:18.067293+0000","last_undegraded":"2024-04-22T01:56:18.067293+0000","last_fullsized":"2024-04-22T01:56:18.067293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.386865+0000","last_change":"2024-04-22T01:55:44.499032+0000","last_active":"2024-04-22T01:55:59.386865+0000","last_peered":"2024-04-22T01:55:59.386865+0000","last_clean":"2024-04-22T01:55:59.386865+0000","last_became_active":"2024-04-22T01:55:44.493349+0000","last_became_peered":"2024-04-22T01:55:44.493349+0000","last_unstale":"2024-04-22T01:56:14.023965+0000","last_undegraded":"2024-04-22T01:55:59.386865+0000","last_fullsized":"2024-04-22T01:55:59.386865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'8","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.061816+0000","last_change":"2024-04-22T01:56:16.179472+0000","last_active":"2024-04-22T01:56:17.061816+0000","last_peered":"2024-04-22T01:56:17.061816+0000","last_clean":"2024-04-22T01:56:17.061816+0000","last_became_active":"2024-04-22T01:56:16.170620+0000","last_became_peered":"2024-04-22T01:56:16.170620+0000","last_unstale":"2024-04-22T01:56:17.061816+0000","last_undegraded":"2024-04-22T01:56:17.061816+0000","last_fullsized":"2024-04-22T01:56:17.061816+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"25'7","reported_seq":34,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.795000+0000","last_change":"2024-04-22T01:55:59.442173+0000","last_active":"2024-04-22T01:55:59.795000+0000","last_peered":"2024-04-22T01:55:59.795000+0000","last_clean":"2024-04-22T01:55:59.795000+0000","last_became_active":"2024-04-22T01:55:59.435623+0000","last_became_peered":"2024-04-22T01:55:59.435623+0000","last_unstale":"2024-04-22T01:56:14.023967+0000","last_undegraded":"2024-04-22T01:55:59.795000+0000","last_fullsized":"2024-04-22T01:55:59.795000+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"25'7","reported_seq":38,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.756631+0000","last_change":"2024-04-22T01:55:59.629702+0000","last_active":"2024-04-22T01:55:59.756631+0000","last_peered":"2024-04-22T01:55:59.756631+0000","last_clean":"2024-04-22T01:55:59.756631+0000","last_became_active":"2024-04-22T01:55:59.453121+0000","last_became_peered":"2024-04-22T01:55:59.453121+0000","last_unstale":"2024-04-22T01:56:14.023969+0000","last_undegraded":"2024-04-22T01:55:59.756631+0000","last_fullsized":"2024-04-22T01:55:59.756631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'16","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.071027+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:56:18.071027+0000","last_peered":"2024-04-22T01:56:18.071027+0000","last_clean":"2024-04-22T01:56:18.071027+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:56:18.071027+0000","last_undegraded":"2024-04-22T01:56:18.071027+0000","last_fullsized":"2024-04-22T01:56:18.071027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":192,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.051297+0000","last_change":"2024-04-22T01:56:16.145535+0000","last_active":"2024-04-22T01:56:17.051297+0000","last_peered":"2024-04-22T01:56:17.051297+0000","last_clean":"2024-04-22T01:56:17.051297+0000","last_became_active":"2024-04-22T01:56:16.138886+0000","last_became_peered":"2024-04-22T01:56:16.138886+0000","last_unstale":"2024-04-22T01:56:17.051297+0000","last_undegraded":"2024-04-22T01:56:17.051297+0000","last_fullsized":"2024-04-22T01:56:17.051297+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.068122+0000","last_change":"2024-04-22T01:56:16.170099+0000","last_active":"2024-04-22T01:56:18.068122+0000","last_peered":"2024-04-22T01:56:18.068122+0000","last_clean":"2024-04-22T01:56:18.068122+0000","last_became_active":"2024-04-22T01:56:16.163854+0000","last_became_peered":"2024-04-22T01:56:16.163854+0000","last_unstale":"2024-04-22T01:56:18.068122+0000","last_undegraded":"2024-04-22T01:56:18.068122+0000","last_fullsized":"2024-04-22T01:56:18.068122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'16","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.059231+0000","last_change":"2024-04-22T01:56:16.277560+0000","last_active":"2024-04-22T01:56:18.059231+0000","last_peered":"2024-04-22T01:56:18.059231+0000","last_clean":"2024-04-22T01:56:18.059231+0000","last_became_active":"2024-04-22T01:56:16.140886+0000","last_became_peered":"2024-04-22T01:56:16.140886+0000","last_unstale":"2024-04-22T01:56:18.059231+0000","last_undegraded":"2024-04-22T01:56:18.059231+0000","last_fullsized":"2024-04-22T01:56:18.059231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":44,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.453870+0000","last_change":"2024-04-22T01:55:59.455186+0000","last_active":"2024-04-22T01:55:59.453870+0000","last_peered":"2024-04-22T01:55:59.453870+0000","last_clean":"2024-04-22T01:55:46.214037+0000","last_became_active":"2024-04-22T01:55:59.449321+0000","last_became_peered":"2024-04-22T01:55:59.449321+0000","last_unstale":"2024-04-22T01:56:14.023972+0000","last_undegraded":"2024-04-22T01:55:59.453870+0000","last_fullsized":"2024-04-22T01:55:59.453870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"25'8","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.057000+0000","last_change":"2024-04-22T01:56:16.132726+0000","last_active":"2024-04-22T01:56:17.057000+0000","last_peered":"2024-04-22T01:56:17.057000+0000","last_clean":"2024-04-22T01:56:17.057000+0000","last_became_active":"2024-04-22T01:56:16.127772+0000","last_became_peered":"2024-04-22T01:56:16.127772+0000","last_unstale":"2024-04-22T01:56:17.057000+0000","last_undegraded":"2024-04-22T01:56:17.057000+0000","last_fullsized":"2024-04-22T01:56:17.057000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.070601+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:56:18.070601+0000","last_peered":"2024-04-22T01:56:18.070601+0000","last_clean":"2024-04-22T01:56:18.070601+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:56:18.070601+0000","last_undegraded":"2024-04-22T01:56:18.070601+0000","last_fullsized":"2024-04-22T01:56:18.070601+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"28'16","reported_seq":58,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.064142+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:56:17.064142+0000","last_peered":"2024-04-22T01:56:17.064142+0000","last_clean":"2024-04-22T01:56:17.064142+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:56:17.064142+0000","last_undegraded":"2024-04-22T01:56:17.064142+0000","last_fullsized":"2024-04-22T01:56:17.064142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":44,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-04-22T01:55:59.455988+0000","last_change":"2024-04-22T01:55:59.457416+0000","last_active":"2024-04-22T01:55:59.455988+0000","last_peered":"2024-04-22T01:55:59.455988+0000","last_clean":"2024-04-22T01:55:46.220728+0000","last_became_active":"2024-04-22T01:55:59.440651+0000","last_became_peered":"2024-04-22T01:55:59.440651+0000","last_unstale":"2024-04-22T01:56:14.023984+0000","last_undegraded":"2024-04-22T01:55:59.455988+0000","last_fullsized":"2024-04-22T01:55:59.455988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.165174+0000","last_change":"2024-04-22T01:56:16.167034+0000","last_active":"2024-04-22T01:56:16.165174+0000","last_peered":"2024-04-22T01:56:16.165174+0000","last_clean":"2024-04-22T01:55:59.913034+0000","last_became_active":"2024-04-22T01:56:16.159121+0000","last_became_peered":"2024-04-22T01:56:16.159121+0000","last_unstale":"2024-04-22T01:56:16.165174+0000","last_undegraded":"2024-04-22T01:56:16.165174+0000","last_fullsized":"2024-04-22T01:56:16.165174+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"28'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069657+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:56:18.069657+0000","last_peered":"2024-04-22T01:56:18.069657+0000","last_clean":"2024-04-22T01:56:18.069657+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:56:18.069657+0000","last_undegraded":"2024-04-22T01:56:18.069657+0000","last_fullsized":"2024-04-22T01:56:18.069657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.161009+0000","last_change":"2024-04-22T01:56:16.162725+0000","last_active":"2024-04-22T01:56:16.161009+0000","last_peered":"2024-04-22T01:56:16.161009+0000","last_clean":"2024-04-22T01:55:59.377500+0000","last_became_active":"2024-04-22T01:56:16.156228+0000","last_became_peered":"2024-04-22T01:56:16.156228+0000","last_unstale":"2024-04-22T01:56:16.161009+0000","last_undegraded":"2024-04-22T01:56:16.161009+0000","last_fullsized":"2024-04-22T01:56:16.161009+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.147048+0000","last_change":"2024-04-22T01:56:16.148704+0000","last_active":"2024-04-22T01:56:16.147048+0000","last_peered":"2024-04-22T01:56:16.147048+0000","last_clean":"2024-04-22T01:55:59.371075+0000","last_became_active":"2024-04-22T01:56:16.142439+0000","last_became_peered":"2024-04-22T01:56:16.142439+0000","last_unstale":"2024-04-22T01:56:16.147048+0000","last_undegraded":"2024-04-22T01:56:16.147048+0000","last_fullsized":"2024-04-22T01:56:16.147048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.060838+0000","last_change":"2024-04-22T01:56:16.165211+0000","last_active":"2024-04-22T01:56:17.060838+0000","last_peered":"2024-04-22T01:56:17.060838+0000","last_clean":"2024-04-22T01:56:17.060838+0000","last_became_active":"2024-04-22T01:56:16.155541+0000","last_became_peered":"2024-04-22T01:56:16.155541+0000","last_unstale":"2024-04-22T01:56:17.060838+0000","last_undegraded":"2024-04-22T01:56:17.060838+0000","last_fullsized":"2024-04-22T01:56:17.060838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.063441+0000","last_change":"2024-04-22T01:56:16.165084+0000","last_active":"2024-04-22T01:56:17.063441+0000","last_peered":"2024-04-22T01:56:17.063441+0000","last_clean":"2024-04-22T01:56:17.063441+0000","last_became_active":"2024-04-22T01:56:16.158792+0000","last_became_peered":"2024-04-22T01:56:16.158792+0000","last_unstale":"2024-04-22T01:56:17.063441+0000","last_undegraded":"2024-04-22T01:56:17.063441+0000","last_fullsized":"2024-04-22T01:56:17.063441+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.188165+0000","last_change":"2024-04-22T01:56:16.189849+0000","last_active":"2024-04-22T01:56:16.188165+0000","last_peered":"2024-04-22T01:56:16.188165+0000","last_clean":"2024-04-22T01:55:59.386495+0000","last_became_active":"2024-04-22T01:56:16.183739+0000","last_became_peered":"2024-04-22T01:56:16.183739+0000","last_unstale":"2024-04-22T01:56:16.188165+0000","last_undegraded":"2024-04-22T01:56:16.188165+0000","last_fullsized":"2024-04-22T01:56:16.188165+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.061099+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:56:16.061099+0000","last_peered":"2024-04-22T01:56:16.061099+0000","last_clean":"2024-04-22T01:56:16.061099+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:56:16.061099+0000","last_undegraded":"2024-04-22T01:56:16.061099+0000","last_fullsized":"2024-04-22T01:56:16.061099+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069950+0000","last_change":"2024-04-22T01:56:16.171487+0000","last_active":"2024-04-22T01:56:18.069950+0000","last_peered":"2024-04-22T01:56:18.069950+0000","last_clean":"2024-04-22T01:56:18.069950+0000","last_became_active":"2024-04-22T01:56:16.165037+0000","last_became_peered":"2024-04-22T01:56:16.165037+0000","last_unstale":"2024-04-22T01:56:18.069950+0000","last_undegraded":"2024-04-22T01:56:18.069950+0000","last_fullsized":"2024-04-22T01:56:18.069950+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142,"ondisk_log_size":142,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27632,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":827392,"data_stored":1721095,"data_compressed":83889,"data_compressed_allocated":565248,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27592,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":720896,"data_stored":1521242,"data_compressed":73761,"data_compressed_allocated":466944,"data_compressed_original":1425408,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27572,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":700416,"data_stored":999655,"data_compressed":47196,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":499712,"data_stored":1117207,"data_compressed":53533,"data_compressed_allocated":270336,"data_compressed_original":1032192,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:56:19.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:56:19.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.828+0000 7f1dc6314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:19.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:19.834+0000 7f1dc6314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:20.089 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:56:20.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":64,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1713750975,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"stale+active+clean","count":6}],"num_pgs":25,"num_pools":3,"num_objects":13,"data_bytes":1253920,"bytes_used":112168960,"bytes_avail":386434887680,"bytes_total":386547056640},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-22T01:55:16.004499+0000","services":{}},"progress_events":{}} 2024-04-22T01:56:20.102 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-22T01:56:20.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:56:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:20.207+0000 7f9ee2f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:20.217+0000 7f9ee2f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:20.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 160 144 0.0261596 0 - 4.36373 2024-04-22T01:56:20.400 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:56:20.401 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:56:20.415 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-04-22T01:56:20.191356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1770016,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":281,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110340,"kb_used_data":2852,"kb_used_omap":0,"kb_used_meta":107456,"kb_avail":377377020,"statfs":{"total":386547056640,"available":386434068480,"internally_reserved":0,"allocated":2920448,"data_stored":5638652,"data_compressed":271875,"data_compressed_allocated":1892352,"data_compressed_original":5259264,"omap_allocated":0,"internal_metadata":110034944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":516096,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.160048"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.069984+0000","last_change":"2024-04-22T01:56:16.175931+0000","last_active":"2024-04-22T01:56:19.069984+0000","last_peered":"2024-04-22T01:56:19.069984+0000","last_clean":"2024-04-22T01:56:19.069984+0000","last_became_active":"2024-04-22T01:56:16.169290+0000","last_became_peered":"2024-04-22T01:56:16.169290+0000","last_unstale":"2024-04-22T01:56:19.069984+0000","last_undegraded":"2024-04-22T01:56:19.069984+0000","last_fullsized":"2024-04-22T01:56:19.069984+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.065226+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:56:16.065226+0000","last_peered":"2024-04-22T01:56:16.065226+0000","last_clean":"2024-04-22T01:56:16.065226+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:56:16.065226+0000","last_undegraded":"2024-04-22T01:56:16.065226+0000","last_fullsized":"2024-04-22T01:56:16.065226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'16","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.067293+0000","last_change":"2024-04-22T01:56:16.167448+0000","last_active":"2024-04-22T01:56:18.067293+0000","last_peered":"2024-04-22T01:56:18.067293+0000","last_clean":"2024-04-22T01:56:18.067293+0000","last_became_active":"2024-04-22T01:56:16.159250+0000","last_became_peered":"2024-04-22T01:56:16.159250+0000","last_unstale":"2024-04-22T01:56:18.067293+0000","last_undegraded":"2024-04-22T01:56:18.067293+0000","last_fullsized":"2024-04-22T01:56:18.067293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.074977+0000","last_change":"2024-04-22T01:56:16.221546+0000","last_active":"2024-04-22T01:56:19.074977+0000","last_peered":"2024-04-22T01:56:19.074977+0000","last_clean":"2024-04-22T01:56:19.074977+0000","last_became_active":"2024-04-22T01:56:16.217388+0000","last_became_peered":"2024-04-22T01:56:16.217388+0000","last_unstale":"2024-04-22T01:56:19.074977+0000","last_undegraded":"2024-04-22T01:56:19.074977+0000","last_fullsized":"2024-04-22T01:56:19.074977+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'8","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.061816+0000","last_change":"2024-04-22T01:56:16.179472+0000","last_active":"2024-04-22T01:56:17.061816+0000","last_peered":"2024-04-22T01:56:17.061816+0000","last_clean":"2024-04-22T01:56:17.061816+0000","last_became_active":"2024-04-22T01:56:16.170620+0000","last_became_peered":"2024-04-22T01:56:16.170620+0000","last_unstale":"2024-04-22T01:56:17.061816+0000","last_undegraded":"2024-04-22T01:56:17.061816+0000","last_fullsized":"2024-04-22T01:56:17.061816+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"25'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.074643+0000","last_change":"2024-04-22T01:56:16.196294+0000","last_active":"2024-04-22T01:56:19.074643+0000","last_peered":"2024-04-22T01:56:19.074643+0000","last_clean":"2024-04-22T01:56:19.074643+0000","last_became_active":"2024-04-22T01:56:16.191541+0000","last_became_peered":"2024-04-22T01:56:16.191541+0000","last_unstale":"2024-04-22T01:56:19.074643+0000","last_undegraded":"2024-04-22T01:56:19.074643+0000","last_fullsized":"2024-04-22T01:56:19.074643+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"25'8","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.062391+0000","last_change":"2024-04-22T01:56:16.218648+0000","last_active":"2024-04-22T01:56:19.062391+0000","last_peered":"2024-04-22T01:56:19.062391+0000","last_clean":"2024-04-22T01:56:19.062391+0000","last_became_active":"2024-04-22T01:56:16.208200+0000","last_became_peered":"2024-04-22T01:56:16.208200+0000","last_unstale":"2024-04-22T01:56:19.062391+0000","last_undegraded":"2024-04-22T01:56:19.062391+0000","last_fullsized":"2024-04-22T01:56:19.062391+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'16","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.071027+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:56:18.071027+0000","last_peered":"2024-04-22T01:56:18.071027+0000","last_clean":"2024-04-22T01:56:18.071027+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:56:18.071027+0000","last_undegraded":"2024-04-22T01:56:18.071027+0000","last_fullsized":"2024-04-22T01:56:18.071027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":192,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.051297+0000","last_change":"2024-04-22T01:56:16.145535+0000","last_active":"2024-04-22T01:56:17.051297+0000","last_peered":"2024-04-22T01:56:17.051297+0000","last_clean":"2024-04-22T01:56:17.051297+0000","last_became_active":"2024-04-22T01:56:16.138886+0000","last_became_peered":"2024-04-22T01:56:16.138886+0000","last_unstale":"2024-04-22T01:56:17.051297+0000","last_undegraded":"2024-04-22T01:56:17.051297+0000","last_fullsized":"2024-04-22T01:56:17.051297+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.068122+0000","last_change":"2024-04-22T01:56:16.170099+0000","last_active":"2024-04-22T01:56:18.068122+0000","last_peered":"2024-04-22T01:56:18.068122+0000","last_clean":"2024-04-22T01:56:18.068122+0000","last_became_active":"2024-04-22T01:56:16.163854+0000","last_became_peered":"2024-04-22T01:56:16.163854+0000","last_unstale":"2024-04-22T01:56:18.068122+0000","last_undegraded":"2024-04-22T01:56:18.068122+0000","last_fullsized":"2024-04-22T01:56:18.068122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'16","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.059231+0000","last_change":"2024-04-22T01:56:16.277560+0000","last_active":"2024-04-22T01:56:18.059231+0000","last_peered":"2024-04-22T01:56:18.059231+0000","last_clean":"2024-04-22T01:56:18.059231+0000","last_became_active":"2024-04-22T01:56:16.140886+0000","last_became_peered":"2024-04-22T01:56:16.140886+0000","last_unstale":"2024-04-22T01:56:18.059231+0000","last_undegraded":"2024-04-22T01:56:18.059231+0000","last_fullsized":"2024-04-22T01:56:18.059231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.077159+0000","last_change":"2024-04-22T01:56:16.235115+0000","last_active":"2024-04-22T01:56:19.077159+0000","last_peered":"2024-04-22T01:56:19.077159+0000","last_clean":"2024-04-22T01:56:19.077159+0000","last_became_active":"2024-04-22T01:56:16.231090+0000","last_became_peered":"2024-04-22T01:56:16.231090+0000","last_unstale":"2024-04-22T01:56:19.077159+0000","last_undegraded":"2024-04-22T01:56:19.077159+0000","last_fullsized":"2024-04-22T01:56:19.077159+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"25'8","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.057000+0000","last_change":"2024-04-22T01:56:16.132726+0000","last_active":"2024-04-22T01:56:17.057000+0000","last_peered":"2024-04-22T01:56:17.057000+0000","last_clean":"2024-04-22T01:56:17.057000+0000","last_became_active":"2024-04-22T01:56:16.127772+0000","last_became_peered":"2024-04-22T01:56:16.127772+0000","last_unstale":"2024-04-22T01:56:17.057000+0000","last_undegraded":"2024-04-22T01:56:17.057000+0000","last_fullsized":"2024-04-22T01:56:17.057000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.070601+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:56:18.070601+0000","last_peered":"2024-04-22T01:56:18.070601+0000","last_clean":"2024-04-22T01:56:18.070601+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:56:18.070601+0000","last_undegraded":"2024-04-22T01:56:18.070601+0000","last_fullsized":"2024-04-22T01:56:18.070601+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"28'16","reported_seq":58,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.064142+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:56:17.064142+0000","last_peered":"2024-04-22T01:56:17.064142+0000","last_clean":"2024-04-22T01:56:17.064142+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:56:17.064142+0000","last_undegraded":"2024-04-22T01:56:17.064142+0000","last_fullsized":"2024-04-22T01:56:17.064142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.076053+0000","last_change":"2024-04-22T01:56:16.213855+0000","last_active":"2024-04-22T01:56:19.076053+0000","last_peered":"2024-04-22T01:56:19.076053+0000","last_clean":"2024-04-22T01:56:19.076053+0000","last_became_active":"2024-04-22T01:56:16.207768+0000","last_became_peered":"2024-04-22T01:56:16.207768+0000","last_unstale":"2024-04-22T01:56:19.076053+0000","last_undegraded":"2024-04-22T01:56:19.076053+0000","last_fullsized":"2024-04-22T01:56:19.076053+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.165174+0000","last_change":"2024-04-22T01:56:16.167034+0000","last_active":"2024-04-22T01:56:16.165174+0000","last_peered":"2024-04-22T01:56:16.165174+0000","last_clean":"2024-04-22T01:55:59.913034+0000","last_became_active":"2024-04-22T01:56:16.159121+0000","last_became_peered":"2024-04-22T01:56:16.159121+0000","last_unstale":"2024-04-22T01:56:16.165174+0000","last_undegraded":"2024-04-22T01:56:16.165174+0000","last_fullsized":"2024-04-22T01:56:16.165174+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"28'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069657+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:56:18.069657+0000","last_peered":"2024-04-22T01:56:18.069657+0000","last_clean":"2024-04-22T01:56:18.069657+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:56:18.069657+0000","last_undegraded":"2024-04-22T01:56:18.069657+0000","last_fullsized":"2024-04-22T01:56:18.069657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.161009+0000","last_change":"2024-04-22T01:56:16.162725+0000","last_active":"2024-04-22T01:56:16.161009+0000","last_peered":"2024-04-22T01:56:16.161009+0000","last_clean":"2024-04-22T01:55:59.377500+0000","last_became_active":"2024-04-22T01:56:16.156228+0000","last_became_peered":"2024-04-22T01:56:16.156228+0000","last_unstale":"2024-04-22T01:56:16.161009+0000","last_undegraded":"2024-04-22T01:56:16.161009+0000","last_fullsized":"2024-04-22T01:56:16.161009+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.147048+0000","last_change":"2024-04-22T01:56:16.148704+0000","last_active":"2024-04-22T01:56:16.147048+0000","last_peered":"2024-04-22T01:56:16.147048+0000","last_clean":"2024-04-22T01:55:59.371075+0000","last_became_active":"2024-04-22T01:56:16.142439+0000","last_became_peered":"2024-04-22T01:56:16.142439+0000","last_unstale":"2024-04-22T01:56:16.147048+0000","last_undegraded":"2024-04-22T01:56:16.147048+0000","last_fullsized":"2024-04-22T01:56:16.147048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.060838+0000","last_change":"2024-04-22T01:56:16.165211+0000","last_active":"2024-04-22T01:56:17.060838+0000","last_peered":"2024-04-22T01:56:17.060838+0000","last_clean":"2024-04-22T01:56:17.060838+0000","last_became_active":"2024-04-22T01:56:16.155541+0000","last_became_peered":"2024-04-22T01:56:16.155541+0000","last_unstale":"2024-04-22T01:56:17.060838+0000","last_undegraded":"2024-04-22T01:56:17.060838+0000","last_fullsized":"2024-04-22T01:56:17.060838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.063441+0000","last_change":"2024-04-22T01:56:16.165084+0000","last_active":"2024-04-22T01:56:17.063441+0000","last_peered":"2024-04-22T01:56:17.063441+0000","last_clean":"2024-04-22T01:56:17.063441+0000","last_became_active":"2024-04-22T01:56:16.158792+0000","last_became_peered":"2024-04-22T01:56:16.158792+0000","last_unstale":"2024-04-22T01:56:17.063441+0000","last_undegraded":"2024-04-22T01:56:17.063441+0000","last_fullsized":"2024-04-22T01:56:17.063441+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.188165+0000","last_change":"2024-04-22T01:56:16.189849+0000","last_active":"2024-04-22T01:56:16.188165+0000","last_peered":"2024-04-22T01:56:16.188165+0000","last_clean":"2024-04-22T01:55:59.386495+0000","last_became_active":"2024-04-22T01:56:16.183739+0000","last_became_peered":"2024-04-22T01:56:16.183739+0000","last_unstale":"2024-04-22T01:56:16.188165+0000","last_undegraded":"2024-04-22T01:56:16.188165+0000","last_fullsized":"2024-04-22T01:56:16.188165+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.061099+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:56:16.061099+0000","last_peered":"2024-04-22T01:56:16.061099+0000","last_clean":"2024-04-22T01:56:16.061099+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:56:16.061099+0000","last_undegraded":"2024-04-22T01:56:16.061099+0000","last_fullsized":"2024-04-22T01:56:16.061099+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069950+0000","last_change":"2024-04-22T01:56:16.171487+0000","last_active":"2024-04-22T01:56:18.069950+0000","last_peered":"2024-04-22T01:56:18.069950+0000","last_clean":"2024-04-22T01:56:18.069950+0000","last_became_active":"2024-04-22T01:56:16.165037+0000","last_became_peered":"2024-04-22T01:56:16.165037+0000","last_unstale":"2024-04-22T01:56:18.069950+0000","last_undegraded":"2024-04-22T01:56:18.069950+0000","last_fullsized":"2024-04-22T01:56:18.069950+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27632,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":827392,"data_stored":1721095,"data_compressed":83889,"data_compressed_allocated":565248,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27592,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":720896,"data_stored":1521242,"data_compressed":73761,"data_compressed_allocated":466944,"data_compressed_original":1425408,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27572,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":700416,"data_stored":999655,"data_compressed":47196,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":671744,"data_stored":1396660,"data_compressed":67029,"data_compressed_allocated":401408,"data_compressed_original":1294336,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:56:20.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:20.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:20.891+0000 7ff90d217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:20.896+0000 7ff90d217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:21.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 160 144 0.0255651 0 - 4.36373 2024-04-22T01:56:22.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 160 144 0.024997 0 - 4.36373 2024-04-22T01:56:22.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:23.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.110+0000 7fc88459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.116+0000 7fc88459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 160 144 0.0244536 0 - 4.36373 2024-04-22T01:56:23.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:56:23.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.525+0000 7fc706817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.532+0000 7fc706817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:56:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:56:23.728 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-04-22T01:56:22.191892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1770016,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":281,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":110340,"kb_used_data":2852,"kb_used_omap":0,"kb_used_meta":107456,"kb_avail":377377020,"statfs":{"total":386547056640,"available":386434068480,"internally_reserved":0,"allocated":2920448,"data_stored":5638652,"data_compressed":271875,"data_compressed_allocated":1892352,"data_compressed_original":5259264,"omap_allocated":0,"internal_metadata":110034944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":516096,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001482"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.069984+0000","last_change":"2024-04-22T01:56:16.175931+0000","last_active":"2024-04-22T01:56:19.069984+0000","last_peered":"2024-04-22T01:56:19.069984+0000","last_clean":"2024-04-22T01:56:19.069984+0000","last_became_active":"2024-04-22T01:56:16.169290+0000","last_became_peered":"2024-04-22T01:56:16.169290+0000","last_unstale":"2024-04-22T01:56:19.069984+0000","last_undegraded":"2024-04-22T01:56:19.069984+0000","last_fullsized":"2024-04-22T01:56:19.069984+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.065226+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:56:16.065226+0000","last_peered":"2024-04-22T01:56:16.065226+0000","last_clean":"2024-04-22T01:56:16.065226+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:56:16.065226+0000","last_undegraded":"2024-04-22T01:56:16.065226+0000","last_fullsized":"2024-04-22T01:56:16.065226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"28'16","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.067293+0000","last_change":"2024-04-22T01:56:16.167448+0000","last_active":"2024-04-22T01:56:18.067293+0000","last_peered":"2024-04-22T01:56:18.067293+0000","last_clean":"2024-04-22T01:56:18.067293+0000","last_became_active":"2024-04-22T01:56:16.159250+0000","last_became_peered":"2024-04-22T01:56:16.159250+0000","last_unstale":"2024-04-22T01:56:18.067293+0000","last_undegraded":"2024-04-22T01:56:18.067293+0000","last_fullsized":"2024-04-22T01:56:18.067293+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.074977+0000","last_change":"2024-04-22T01:56:16.221546+0000","last_active":"2024-04-22T01:56:19.074977+0000","last_peered":"2024-04-22T01:56:19.074977+0000","last_clean":"2024-04-22T01:56:19.074977+0000","last_became_active":"2024-04-22T01:56:16.217388+0000","last_became_peered":"2024-04-22T01:56:16.217388+0000","last_unstale":"2024-04-22T01:56:19.074977+0000","last_undegraded":"2024-04-22T01:56:19.074977+0000","last_fullsized":"2024-04-22T01:56:19.074977+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'8","reported_seq":46,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.061816+0000","last_change":"2024-04-22T01:56:16.179472+0000","last_active":"2024-04-22T01:56:17.061816+0000","last_peered":"2024-04-22T01:56:17.061816+0000","last_clean":"2024-04-22T01:56:17.061816+0000","last_became_active":"2024-04-22T01:56:16.170620+0000","last_became_peered":"2024-04-22T01:56:16.170620+0000","last_unstale":"2024-04-22T01:56:17.061816+0000","last_undegraded":"2024-04-22T01:56:17.061816+0000","last_fullsized":"2024-04-22T01:56:17.061816+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"25'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.074643+0000","last_change":"2024-04-22T01:56:16.196294+0000","last_active":"2024-04-22T01:56:19.074643+0000","last_peered":"2024-04-22T01:56:19.074643+0000","last_clean":"2024-04-22T01:56:19.074643+0000","last_became_active":"2024-04-22T01:56:16.191541+0000","last_became_peered":"2024-04-22T01:56:16.191541+0000","last_unstale":"2024-04-22T01:56:19.074643+0000","last_undegraded":"2024-04-22T01:56:19.074643+0000","last_fullsized":"2024-04-22T01:56:19.074643+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"25'8","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.062391+0000","last_change":"2024-04-22T01:56:16.218648+0000","last_active":"2024-04-22T01:56:19.062391+0000","last_peered":"2024-04-22T01:56:19.062391+0000","last_clean":"2024-04-22T01:56:19.062391+0000","last_became_active":"2024-04-22T01:56:16.208200+0000","last_became_peered":"2024-04-22T01:56:16.208200+0000","last_unstale":"2024-04-22T01:56:19.062391+0000","last_undegraded":"2024-04-22T01:56:19.062391+0000","last_fullsized":"2024-04-22T01:56:19.062391+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"28'16","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.071027+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:56:18.071027+0000","last_peered":"2024-04-22T01:56:18.071027+0000","last_clean":"2024-04-22T01:56:18.071027+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:56:18.071027+0000","last_undegraded":"2024-04-22T01:56:18.071027+0000","last_fullsized":"2024-04-22T01:56:18.071027+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":192,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.051297+0000","last_change":"2024-04-22T01:56:16.145535+0000","last_active":"2024-04-22T01:56:17.051297+0000","last_peered":"2024-04-22T01:56:17.051297+0000","last_clean":"2024-04-22T01:56:17.051297+0000","last_became_active":"2024-04-22T01:56:16.138886+0000","last_became_peered":"2024-04-22T01:56:16.138886+0000","last_unstale":"2024-04-22T01:56:17.051297+0000","last_undegraded":"2024-04-22T01:56:17.051297+0000","last_fullsized":"2024-04-22T01:56:17.051297+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.068122+0000","last_change":"2024-04-22T01:56:16.170099+0000","last_active":"2024-04-22T01:56:18.068122+0000","last_peered":"2024-04-22T01:56:18.068122+0000","last_clean":"2024-04-22T01:56:18.068122+0000","last_became_active":"2024-04-22T01:56:16.163854+0000","last_became_peered":"2024-04-22T01:56:16.163854+0000","last_unstale":"2024-04-22T01:56:18.068122+0000","last_undegraded":"2024-04-22T01:56:18.068122+0000","last_fullsized":"2024-04-22T01:56:18.068122+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'16","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.059231+0000","last_change":"2024-04-22T01:56:16.277560+0000","last_active":"2024-04-22T01:56:18.059231+0000","last_peered":"2024-04-22T01:56:18.059231+0000","last_clean":"2024-04-22T01:56:18.059231+0000","last_became_active":"2024-04-22T01:56:16.140886+0000","last_became_peered":"2024-04-22T01:56:16.140886+0000","last_unstale":"2024-04-22T01:56:18.059231+0000","last_undegraded":"2024-04-22T01:56:18.059231+0000","last_fullsized":"2024-04-22T01:56:18.059231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.077159+0000","last_change":"2024-04-22T01:56:16.235115+0000","last_active":"2024-04-22T01:56:19.077159+0000","last_peered":"2024-04-22T01:56:19.077159+0000","last_clean":"2024-04-22T01:56:19.077159+0000","last_became_active":"2024-04-22T01:56:16.231090+0000","last_became_peered":"2024-04-22T01:56:16.231090+0000","last_unstale":"2024-04-22T01:56:19.077159+0000","last_undegraded":"2024-04-22T01:56:19.077159+0000","last_fullsized":"2024-04-22T01:56:19.077159+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"25'8","reported_seq":55,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.057000+0000","last_change":"2024-04-22T01:56:16.132726+0000","last_active":"2024-04-22T01:56:17.057000+0000","last_peered":"2024-04-22T01:56:17.057000+0000","last_clean":"2024-04-22T01:56:17.057000+0000","last_became_active":"2024-04-22T01:56:16.127772+0000","last_became_peered":"2024-04-22T01:56:16.127772+0000","last_unstale":"2024-04-22T01:56:17.057000+0000","last_undegraded":"2024-04-22T01:56:17.057000+0000","last_fullsized":"2024-04-22T01:56:17.057000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.070601+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:56:18.070601+0000","last_peered":"2024-04-22T01:56:18.070601+0000","last_clean":"2024-04-22T01:56:18.070601+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:56:18.070601+0000","last_undegraded":"2024-04-22T01:56:18.070601+0000","last_fullsized":"2024-04-22T01:56:18.070601+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"28'16","reported_seq":58,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.064142+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:56:17.064142+0000","last_peered":"2024-04-22T01:56:17.064142+0000","last_clean":"2024-04-22T01:56:17.064142+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:56:17.064142+0000","last_undegraded":"2024-04-22T01:56:17.064142+0000","last_fullsized":"2024-04-22T01:56:17.064142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-22T01:56:19.076053+0000","last_change":"2024-04-22T01:56:16.213855+0000","last_active":"2024-04-22T01:56:19.076053+0000","last_peered":"2024-04-22T01:56:19.076053+0000","last_clean":"2024-04-22T01:56:19.076053+0000","last_became_active":"2024-04-22T01:56:16.207768+0000","last_became_peered":"2024-04-22T01:56:16.207768+0000","last_unstale":"2024-04-22T01:56:19.076053+0000","last_undegraded":"2024-04-22T01:56:19.076053+0000","last_fullsized":"2024-04-22T01:56:19.076053+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"25'16","reported_seq":47,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.165174+0000","last_change":"2024-04-22T01:56:16.167034+0000","last_active":"2024-04-22T01:56:16.165174+0000","last_peered":"2024-04-22T01:56:16.165174+0000","last_clean":"2024-04-22T01:55:59.913034+0000","last_became_active":"2024-04-22T01:56:16.159121+0000","last_became_peered":"2024-04-22T01:56:16.159121+0000","last_unstale":"2024-04-22T01:56:16.165174+0000","last_undegraded":"2024-04-22T01:56:16.165174+0000","last_fullsized":"2024-04-22T01:56:16.165174+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"28'8","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069657+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:56:18.069657+0000","last_peered":"2024-04-22T01:56:18.069657+0000","last_clean":"2024-04-22T01:56:18.069657+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:56:18.069657+0000","last_undegraded":"2024-04-22T01:56:18.069657+0000","last_fullsized":"2024-04-22T01:56:18.069657+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.161009+0000","last_change":"2024-04-22T01:56:16.162725+0000","last_active":"2024-04-22T01:56:16.161009+0000","last_peered":"2024-04-22T01:56:16.161009+0000","last_clean":"2024-04-22T01:55:59.377500+0000","last_became_active":"2024-04-22T01:56:16.156228+0000","last_became_peered":"2024-04-22T01:56:16.156228+0000","last_unstale":"2024-04-22T01:56:16.161009+0000","last_undegraded":"2024-04-22T01:56:16.161009+0000","last_fullsized":"2024-04-22T01:56:16.161009+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'8","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.147048+0000","last_change":"2024-04-22T01:56:16.148704+0000","last_active":"2024-04-22T01:56:16.147048+0000","last_peered":"2024-04-22T01:56:16.147048+0000","last_clean":"2024-04-22T01:55:59.371075+0000","last_became_active":"2024-04-22T01:56:16.142439+0000","last_became_peered":"2024-04-22T01:56:16.142439+0000","last_unstale":"2024-04-22T01:56:16.147048+0000","last_undegraded":"2024-04-22T01:56:16.147048+0000","last_fullsized":"2024-04-22T01:56:16.147048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.060838+0000","last_change":"2024-04-22T01:56:16.165211+0000","last_active":"2024-04-22T01:56:17.060838+0000","last_peered":"2024-04-22T01:56:17.060838+0000","last_clean":"2024-04-22T01:56:17.060838+0000","last_became_active":"2024-04-22T01:56:16.155541+0000","last_became_peered":"2024-04-22T01:56:16.155541+0000","last_unstale":"2024-04-22T01:56:17.060838+0000","last_undegraded":"2024-04-22T01:56:17.060838+0000","last_fullsized":"2024-04-22T01:56:17.060838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-22T01:56:17.063441+0000","last_change":"2024-04-22T01:56:16.165084+0000","last_active":"2024-04-22T01:56:17.063441+0000","last_peered":"2024-04-22T01:56:17.063441+0000","last_clean":"2024-04-22T01:56:17.063441+0000","last_became_active":"2024-04-22T01:56:16.158792+0000","last_became_peered":"2024-04-22T01:56:16.158792+0000","last_unstale":"2024-04-22T01:56:17.063441+0000","last_undegraded":"2024-04-22T01:56:17.063441+0000","last_fullsized":"2024-04-22T01:56:17.063441+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.188165+0000","last_change":"2024-04-22T01:56:16.189849+0000","last_active":"2024-04-22T01:56:16.188165+0000","last_peered":"2024-04-22T01:56:16.188165+0000","last_clean":"2024-04-22T01:55:59.386495+0000","last_became_active":"2024-04-22T01:56:16.183739+0000","last_became_peered":"2024-04-22T01:56:16.183739+0000","last_unstale":"2024-04-22T01:56:16.188165+0000","last_undegraded":"2024-04-22T01:56:16.188165+0000","last_fullsized":"2024-04-22T01:56:16.188165+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-04-22T01:56:16.061099+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:56:16.061099+0000","last_peered":"2024-04-22T01:56:16.061099+0000","last_clean":"2024-04-22T01:56:16.061099+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:56:16.061099+0000","last_undegraded":"2024-04-22T01:56:16.061099+0000","last_fullsized":"2024-04-22T01:56:16.061099+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-22T01:56:18.069950+0000","last_change":"2024-04-22T01:56:16.171487+0000","last_active":"2024-04-22T01:56:18.069950+0000","last_peered":"2024-04-22T01:56:18.069950+0000","last_clean":"2024-04-22T01:56:18.069950+0000","last_became_active":"2024-04-22T01:56:16.165037+0000","last_became_peered":"2024-04-22T01:56:16.165037+0000","last_unstale":"2024-04-22T01:56:18.069950+0000","last_undegraded":"2024-04-22T01:56:18.069950+0000","last_fullsized":"2024-04-22T01:56:18.069950+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":144,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27632,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":827392,"data_stored":1721095,"data_compressed":83889,"data_compressed_allocated":565248,"data_compressed_original":1622016,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27592,"kb_used_data":704,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":720896,"data_stored":1521242,"data_compressed":73761,"data_compressed_allocated":466944,"data_compressed_original":1425408,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27572,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344268,"statfs":{"total":96636764160,"available":96608530432,"internally_reserved":0,"allocated":700416,"data_stored":999655,"data_compressed":47196,"data_compressed_allocated":458752,"data_compressed_original":917504,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":26880,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":671744,"data_stored":1396660,"data_compressed":67029,"data_compressed_allocated":401408,"data_compressed_original":1294336,"omap_allocated":0,"internal_metadata":27525120},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:56:23.729 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T01:56:23.729 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:23.729 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:56:23.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:56:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.832+0000 7f4475a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.840+0000 7f4475a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:23.860 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:23.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:56:23.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.964+0000 7fbca699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:23.972+0000 7fbca699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:23.991 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:23.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:56:24.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:24.095+0000 7fbd9e0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:24.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:24.104+0000 7fbd9e0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:24.124 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:24.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:56:24.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:24.249+0000 7f1ae415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:24.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:24.258+0000 7f1ae415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:24.272 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:24.272 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:24.272 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:24.273 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:24.276 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:24.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 160 144 0.0239333 0 - 4.36373 2024-04-22T01:56:25.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:25.330+0000 7fd39e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:25.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:25.335+0000 7fd39e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:25.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 160 144 0.0234347 0 - 4.36373 2024-04-22T01:56:26.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 160 144 0.0229564 0 - 4.36373 2024-04-22T01:56:27.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 160 144 0.0224973 0 - 4.36373 2024-04-22T01:56:27.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:27.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:27.592+0000 7fc308b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:27.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:27.599+0000 7fc308b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:28.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 307 291 0.0445719 0.104403 0.0196669 2.80059 2024-04-22T01:56:29.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 1, 0] 2024-04-22T01:56:29.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:29.277 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:56:29.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:56:29.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 468 452 0.0679006 1.25781 0.131099 1.8388 2024-04-22T01:56:29.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.401+0000 7f2087c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.411+0000 7f2087c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:29.441 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:29.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:56:29.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.601+0000 7f4e0a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.614+0000 7f4e0a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:29.646 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:29.647 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:29.647 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:29.650 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:29.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:56:29.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:29.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.784+0000 7f21ad59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.796+0000 7f21ad59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:29.821 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:29.825 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:29.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:56:29.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.923+0000 7ff688f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:29.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:29.931+0000 7ff688f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:30.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:30.023+0000 7f8399554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:30.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:30.035+0000 7f8399554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:30.054 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:30.057 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:30.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 614 598 0.0881381 1.14062 0.148096 1.41581 2024-04-22T01:56:31.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 770 754 0.109073 1.21875 0.252004 1.14381 2024-04-22T01:56:32.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:32.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:32.338+0000 7f29ddb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:32.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:32.345+0000 7f29ddb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:32.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 943 927 0.131661 1.35156 0.0628328 0.948454 2024-04-22T01:56:33.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 1100 1084 0.15121 1.22656 0.122623 0.82567 2024-04-22T01:56:34.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 1268 1252 0.171581 1.3125 0.0711382 0.72784 2024-04-22T01:56:34.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:34.649+0000 7f33baf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:34.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:34.655+0000 7f33baf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.058 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 1, 0] 2024-04-22T01:56:35.058 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:35.058 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:56:35.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:56:35.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.196+0000 7f7e7726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.208+0000 7f7e7726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:35.292 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:35.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:56:35.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 1429 1413 0.190307 1.25781 0.159097 0.655894 2024-04-22T01:56:35.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.433+0000 7f8aac0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.444+0000 7f8aac0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:35.475 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:35.479 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:35.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:56:35.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.605+0000 7f64bca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.614+0000 7f64bca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.631 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:35.632 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:35.635 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:35.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:56:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.754+0000 7fbd02613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:35.764+0000 7fbd02613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:35.789 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:35.790 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:35.790 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:35.794 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:36.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 1595 1579 0.20906 1.29688 0.11445 0.597273 2024-04-22T01:56:36.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:37.035+0000 7fc1654ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:37.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:37.042+0000 7fc1654ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:37.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:56:37.394623+0000 min lat: 0.0165169 max lat: 16.5699 avg lat: 0.547558 2024-04-22T01:56:37.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:56:37.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 1768 1752 0.228099 1.35156 0.0995233 0.547558 2024-04-22T01:56:38.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 1938 1922 0.246129 1.32812 0.170782 0.507355 2024-04-22T01:56:39.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:39.344+0000 7ffb0166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:39.350+0000 7ffb0166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:39.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 2088 2072 0.261059 1.17188 0.150156 0.478244 2024-04-22T01:56:40.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 2247 2231 0.27663 1.24219 0.0449728 0.451409 2024-04-22T01:56:40.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 1, 0] 2024-04-22T01:56:40.796 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:40.796 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:56:40.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:56:40.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:40.931+0000 7fd062e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:40.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:40.943+0000 7fd062e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:40.976 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:40.978 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:40.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:56:41.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.116+0000 7f5e4ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.129+0000 7f5e4ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.166 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:41.166 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:41.166 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:41.166 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:41.166 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:41.167 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:41.170 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:41.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:56:41.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.314+0000 7f0e4cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.325+0000 7f0e4cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:41.354 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:41.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:56:41.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 2405 2389 0.291593 1.23438 0.0240885 0.428206 2024-04-22T01:56:41.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.469+0000 7f8e266ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.479+0000 7f8e266ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.503 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:56:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:56:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:41.507 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:56:41.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:41.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.691+0000 7f877beed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:41.698+0000 7f877beed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:42.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 2570 2554 0.306936 1.28906 0.139954 0.406852 2024-04-22T01:56:43.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 2747 2731 0.323235 1.38281 0.0506359 0.386361 2024-04-22T01:56:43.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:44.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:44.014+0000 7fb3b8aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:44.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:44.020+0000 7fb3b8aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:44.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 2921 2905 0.338698 1.35938 0.0927084 0.368768 2024-04-22T01:56:45.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 3095 3079 0.353706 1.35938 0.0479472 0.353032 2024-04-22T01:56:46.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:56:46.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:46.350+0000 7f6853b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:46.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:46.358+0000 7f6853b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:46.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 3277 3261 0.369184 1.42188 0.0931088 0.338297 2024-04-22T01:56:46.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 1, 0] 2024-04-22T01:56:46.508 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:56:46.509 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [3, 2, 1, 0] 2024-04-22T01:56:46.509 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:56:46.509 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:56:46.591 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:56:47.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 3324 3308 0.369155 0.367188 0.0848229 0.334808 2024-04-22T01:56:48.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 3324 3308 0.363956 0 - 0.334808 2024-04-22T01:56:48.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:56:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:48.661+0000 7f7941224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:48.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:56:48.667+0000 7f7941224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:49.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 3324 3308 0.358901 0 - 0.334808 2024-04-22T01:56:50.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 3324 3308 0.353984 0 - 0.334808 2024-04-22T01:56:51.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 3324 3308 0.3492 0 - 0.334808 2024-04-22T01:56:51.592 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [3, 2, 1] 2024-04-22T01:56:51.592 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-22T01:56:51.592 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:56:51.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:56:51.595 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:56:51.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:56:51.705 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:56:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:51.708 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:56:52.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 3324 3308 0.344544 0 - 0.334808 2024-04-22T01:56:53.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 3324 3308 0.340011 0 - 0.334808 2024-04-22T01:56:54.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 3324 3308 0.335595 0 - 0.334808 2024-04-22T01:56:55.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 3324 3308 0.331292 0 - 0.334808 2024-04-22T01:56:56.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 3324 3308 0.327099 0 - 0.334808 2024-04-22T01:56:56.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:56:56.805 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:56:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:56:56.807 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:56:57.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:56:57.396791+0000 min lat: 0.0159625 max lat: 16.5699 avg lat: 0.334808 2024-04-22T01:56:57.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:56:57.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 3324 3308 0.32301 0 - 0.334808 2024-04-22T01:56:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:56:58.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 3324 3308 0.319023 0 - 0.334808 2024-04-22T01:56:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:56:58.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:56:59.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 3324 3308 0.315132 0 - 0.334808 2024-04-22T01:57:00.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 3324 3308 0.311335 0 - 0.334808 2024-04-22T01:57:00.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:00.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:00.656+0000 7fcacfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:00.663+0000 7fcacfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:01.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 3450 3434 0.319347 0.0703125 0.191163 0.391134 2024-04-22T01:57:01.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [ 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 271, 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.070391+0000" 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.936 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.070408+0000" 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.070563+0000" 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.937 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.070651+0000" 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.070745+0000" 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.095477+0000" 2024-04-22T01:57:01.938 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 274, 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.088181+0000" 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.088205+0000" 2024-04-22T01:57:01.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.088366+0000" 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.088461+0000" 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.940 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.088547+0000" 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.124509+0000" 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 277, 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.108055+0000" 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.108070+0000" 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.108629+0000" 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.108894+0000" 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.109152+0000" 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.151697+0000" 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 279, 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.944 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.111841+0000" 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.111862+0000" 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.112402+0000" 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.114060+0000" 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.114299+0000" 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.946 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.173984+0000" 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 281, 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.115776+0000" 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.947 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.115785+0000" 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.117613+0000" 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.118077+0000" 2024-04-22T01:57:01.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.119863+0000" 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.193712+0000" 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 282, 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.124859+0000" 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.124870+0000" 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.950 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.125221+0000" 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.126083+0000" 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.951 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.126844+0000" 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.211727+0000" 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.952 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 284, 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.134699+0000" 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.134723+0000" 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.136410+0000" 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.953 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.136746+0000" 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.136907+0000" 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.954 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.230131+0000" 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 287, 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.141687+0000" 2024-04-22T01:57:01.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.141708+0000" 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.141974+0000" 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.956 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.143291+0000" 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.143482+0000" 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.249834+0000" 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 302, 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.301360+0000" 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.301378+0000" 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.301501+0000" 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.959 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.301590+0000" 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.301705+0000" 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.960 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.327633+0000" 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 303, 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.307974+0000" 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.961 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.307991+0000" 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.308395+0000" 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.962 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.308602+0000" 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.308784+0000" 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.347163+0000" 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 305, 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.315041+0000" 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.964 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.315057+0000" 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.315204+0000" 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.315288+0000" 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.965 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.315480+0000" 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.368472+0000" 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 306, 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.319095+0000" 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.319109+0000" 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.967 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.319329+0000" 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.319486+0000" 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.968 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.319616+0000" 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.389239+0000" 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 308, 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.334574+0000" 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.334599+0000" 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.970 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.334862+0000" 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.335126+0000" 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.335340+0000" 2024-04-22T01:57:01.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.421175+0000" 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 309, 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.337532+0000" 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.337546+0000" 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.338068+0000" 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.973 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.338244+0000" 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.338423+0000" 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.429739+0000" 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 311, 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.345261+0000" 2024-04-22T01:57:01.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.345279+0000" 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.345448+0000" 2024-04-22T01:57:01.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.345559+0000" 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.345699+0000" 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.451465+0000" 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 312, 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.978 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.352486+0000" 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.352499+0000" 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.352695+0000" 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.979 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.352886+0000" 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.353916+0000" 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:01.468430+0000" 2024-04-22T01:57:01.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 208, 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.655881+0000" 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.655905+0000" 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.981 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.656070+0000" 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.656178+0000" 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.982 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.656460+0000" 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.683184+0000" 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.983 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 210, 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.671575+0000" 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.671592+0000" 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.671750+0000" 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.984 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.671874+0000" 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.672194+0000" 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.713852+0000" 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 211, 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.986 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.672342+0000" 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.672349+0000" 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.672486+0000" 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.987 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.672562+0000" 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.673263+0000" 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.737242+0000" 2024-04-22T01:57:01.988 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 214, 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.686675+0000" 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.686690+0000" 2024-04-22T01:57:01.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.686861+0000" 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.686981+0000" 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.990 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.687202+0000" 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.762317+0000" 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.991 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 215, 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.690271+0000" 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.690301+0000" 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.992 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.690494+0000" 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.690582+0000" 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.993 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.691114+0000" 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.784710+0000" 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 217, 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.697714+0000" 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.697731+0000" 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.697904+0000" 2024-04-22T01:57:01.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.698011+0000" 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.698283+0000" 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.996 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.809587+0000" 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 218, 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.709094+0000" 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.709114+0000" 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.998 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.709584+0000" 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.709827+0000" 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.712430+0000" 2024-04-22T01:57:01.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.815392+0000" 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 220, 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:02.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.712731+0000" 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.712745+0000" 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.713201+0000" 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.001 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.713300+0000" 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:02.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.715042+0000" 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:00.843322+0000" 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:57:02.003 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:57:02.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T01:57:02.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:02.152+0000 7fbc8375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:02.168+0000 7fbc8375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 3610 3594 0.330294 1.25 0.0713554 0.378186 2024-04-22T01:57:03.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:03.169+0000 7f4637cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:03.175+0000 7f4637cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.267 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T01:57:03.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T01:57:03.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 3759 3743 0.339987 1.16406 0.126873 0.367406 2024-04-22T01:57:03.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:03.421+0000 7f91732ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:03.434+0000 7f91732ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.270 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T01:57:04.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T01:57:04.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 3901 3885 0.34883 1.10938 0.0628284 0.358028 2024-04-22T01:57:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:04.443+0000 7fa1f6aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:04.457+0000 7fa1f6aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T01:57:05.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T01:57:05.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 4053 4037 0.358359 1.1875 0.0649484 0.348558 2024-04-22T01:57:05.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:05.430+0000 7fb74019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:05.442+0000 7fb74019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:05.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:05.611+0000 7f5f4959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:05.621+0000 7f5f4959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T01:57:06.301 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T01:57:06.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:57:06.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 4200 4184 0.367235 1.14844 0.133891 0.340069 2024-04-22T01:57:06.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:06.435+0000 7f91152ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:06.444+0000 7f91152ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:06.669 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:57:06.670 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:57:06.688 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-22T01:57:06.283677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31138336,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3866,"num_write_kb":31726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3816,"ondisk_log_size":3816,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":202352,"kb_used_data":44688,"kb_used_omap":0,"kb_used_meta":157632,"kb_avail":377285008,"statfs":{"total":386547056640,"available":386339848192,"internally_reserved":0,"allocated":45760512,"data_stored":91068496,"data_compressed":4676934,"data_compressed_allocated":44568576,"data_compressed_original":90619904,"omap_allocated":0,"internal_metadata":161415168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8085504,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":987,"num_write_kb":7896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082963"},"pg_stats":[{"pgid":"3.f","version":"35'215","reported_seq":280,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.368513+0000","last_change":"2024-04-22T01:57:00.436335+0000","last_active":"2024-04-22T01:57:03.368513+0000","last_peered":"2024-04-22T01:57:03.368513+0000","last_clean":"2024-04-22T01:57:03.368513+0000","last_became_active":"2024-04-22T01:57:00.426790+0000","last_became_peered":"2024-04-22T01:57:00.426790+0000","last_unstale":"2024-04-22T01:57:03.368513+0000","last_undegraded":"2024-04-22T01:57:03.368513+0000","last_fullsized":"2024-04-22T01:57:03.368513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"34'245","reported_seq":295,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.976100+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:57:02.976100+0000","last_peered":"2024-04-22T01:57:02.976100+0000","last_clean":"2024-04-22T01:57:02.976100+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:57:02.976100+0000","last_undegraded":"2024-04-22T01:57:02.976100+0000","last_fullsized":"2024-04-22T01:57:02.976100+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2015232,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":246,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"34'216","reported_seq":266,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.315969+0000","last_change":"2024-04-22T01:57:00.396628+0000","last_active":"2024-04-22T01:57:03.315969+0000","last_peered":"2024-04-22T01:57:03.315969+0000","last_clean":"2024-04-22T01:57:03.315969+0000","last_became_active":"2024-04-22T01:57:00.392365+0000","last_became_peered":"2024-04-22T01:57:00.392365+0000","last_unstale":"2024-04-22T01:57:03.315969+0000","last_undegraded":"2024-04-22T01:57:03.315969+0000","last_fullsized":"2024-04-22T01:57:03.315969+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"35'303","reported_seq":347,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.555870+0000","last_change":"2024-04-22T01:57:00.465981+0000","last_active":"2024-04-22T01:57:03.555870+0000","last_peered":"2024-04-22T01:57:03.555870+0000","last_clean":"2024-04-22T01:57:03.555870+0000","last_became_active":"2024-04-22T01:57:00.458041+0000","last_became_peered":"2024-04-22T01:57:00.458041+0000","last_unstale":"2024-04-22T01:57:03.555870+0000","last_undegraded":"2024-04-22T01:57:03.555870+0000","last_fullsized":"2024-04-22T01:57:03.555870+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"34'199","reported_seq":259,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.922852+0000","last_change":"2024-04-22T01:57:00.553609+0000","last_active":"2024-04-22T01:57:02.922852+0000","last_peered":"2024-04-22T01:57:02.922852+0000","last_clean":"2024-04-22T01:57:02.922852+0000","last_became_active":"2024-04-22T01:57:00.404852+0000","last_became_peered":"2024-04-22T01:57:00.404852+0000","last_unstale":"2024-04-22T01:57:02.922852+0000","last_undegraded":"2024-04-22T01:57:02.922852+0000","last_fullsized":"2024-04-22T01:57:02.922852+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'200","reported_seq":256,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.303339+0000","last_change":"2024-04-22T01:57:00.449671+0000","last_active":"2024-04-22T01:57:03.303339+0000","last_peered":"2024-04-22T01:57:03.303339+0000","last_clean":"2024-04-22T01:57:03.303339+0000","last_became_active":"2024-04-22T01:57:00.445304+0000","last_became_peered":"2024-04-22T01:57:00.445304+0000","last_unstale":"2024-04-22T01:57:03.303339+0000","last_undegraded":"2024-04-22T01:57:03.303339+0000","last_fullsized":"2024-04-22T01:57:03.303339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"34'176","reported_seq":236,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.301038+0000","last_change":"2024-04-22T01:57:00.450117+0000","last_active":"2024-04-22T01:57:03.301038+0000","last_peered":"2024-04-22T01:57:03.301038+0000","last_clean":"2024-04-22T01:57:03.301038+0000","last_became_active":"2024-04-22T01:57:00.439954+0000","last_became_peered":"2024-04-22T01:57:00.439954+0000","last_unstale":"2024-04-22T01:57:03.301038+0000","last_undegraded":"2024-04-22T01:57:03.301038+0000","last_fullsized":"2024-04-22T01:57:03.301038+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"34'272","reported_seq":324,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.308062+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:57:03.308062+0000","last_peered":"2024-04-22T01:57:03.308062+0000","last_clean":"2024-04-22T01:57:03.308062+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:57:03.308062+0000","last_undegraded":"2024-04-22T01:57:03.308062+0000","last_fullsized":"2024-04-22T01:57:03.308062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":208,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.363738+0000","last_change":"2024-04-22T01:57:00.364902+0000","last_active":"2024-04-22T01:57:00.363738+0000","last_peered":"2024-04-22T01:57:00.363738+0000","last_clean":"2024-04-22T01:56:19.101660+0000","last_became_active":"2024-04-22T01:57:00.359940+0000","last_became_peered":"2024-04-22T01:57:00.359940+0000","last_unstale":"2024-04-22T01:57:00.363738+0000","last_undegraded":"2024-04-22T01:57:00.363738+0000","last_fullsized":"2024-04-22T01:57:00.363738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.312018+0000","last_change":"2024-04-22T01:57:00.404788+0000","last_active":"2024-04-22T01:57:03.312018+0000","last_peered":"2024-04-22T01:57:03.312018+0000","last_clean":"2024-04-22T01:57:03.312018+0000","last_became_active":"2024-04-22T01:57:00.399860+0000","last_became_peered":"2024-04-22T01:57:00.399860+0000","last_unstale":"2024-04-22T01:57:03.312018+0000","last_undegraded":"2024-04-22T01:57:03.312018+0000","last_fullsized":"2024-04-22T01:57:03.312018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"34'240","reported_seq":294,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.297876+0000","last_change":"2024-04-22T01:57:00.367651+0000","last_active":"2024-04-22T01:57:03.297876+0000","last_peered":"2024-04-22T01:57:03.297876+0000","last_clean":"2024-04-22T01:57:03.297876+0000","last_became_active":"2024-04-22T01:57:00.362726+0000","last_became_peered":"2024-04-22T01:57:00.362726+0000","last_unstale":"2024-04-22T01:57:03.297876+0000","last_undegraded":"2024-04-22T01:57:03.297876+0000","last_fullsized":"2024-04-22T01:57:03.297876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.309476+0000","last_change":"2024-04-22T01:57:00.415221+0000","last_active":"2024-04-22T01:57:03.309476+0000","last_peered":"2024-04-22T01:57:03.309476+0000","last_clean":"2024-04-22T01:57:03.309476+0000","last_became_active":"2024-04-22T01:57:00.410397+0000","last_became_peered":"2024-04-22T01:57:00.410397+0000","last_unstale":"2024-04-22T01:57:03.309476+0000","last_undegraded":"2024-04-22T01:57:03.309476+0000","last_fullsized":"2024-04-22T01:57:03.309476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"34'191","reported_seq":255,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:01.708391+0000","last_change":"2024-04-22T01:57:00.379491+0000","last_active":"2024-04-22T01:57:01.708391+0000","last_peered":"2024-04-22T01:57:01.708391+0000","last_clean":"2024-04-22T01:57:01.708391+0000","last_became_active":"2024-04-22T01:57:00.373330+0000","last_became_peered":"2024-04-22T01:57:00.373330+0000","last_unstale":"2024-04-22T01:57:01.708391+0000","last_undegraded":"2024-04-22T01:57:01.708391+0000","last_fullsized":"2024-04-22T01:57:01.708391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.314695+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:57:03.314695+0000","last_peered":"2024-04-22T01:57:03.314695+0000","last_clean":"2024-04-22T01:57:03.314695+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:57:03.314695+0000","last_undegraded":"2024-04-22T01:57:03.314695+0000","last_fullsized":"2024-04-22T01:57:03.314695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"34'244","reported_seq":302,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.183241+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:57:03.183241+0000","last_peered":"2024-04-22T01:57:03.183241+0000","last_clean":"2024-04-22T01:57:03.183241+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:57:03.183241+0000","last_undegraded":"2024-04-22T01:57:03.183241+0000","last_fullsized":"2024-04-22T01:57:03.183241+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007040,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":1960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.303795+0000","last_change":"2024-04-22T01:57:00.468645+0000","last_active":"2024-04-22T01:57:03.303795+0000","last_peered":"2024-04-22T01:57:03.303795+0000","last_clean":"2024-04-22T01:57:03.303795+0000","last_became_active":"2024-04-22T01:57:00.462213+0000","last_became_peered":"2024-04-22T01:57:00.462213+0000","last_unstale":"2024-04-22T01:57:03.303795+0000","last_undegraded":"2024-04-22T01:57:03.303795+0000","last_fullsized":"2024-04-22T01:57:03.303795+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"34'151","reported_seq":207,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.885695+0000","last_change":"2024-04-22T01:57:00.571514+0000","last_active":"2024-04-22T01:57:02.885695+0000","last_peered":"2024-04-22T01:57:02.885695+0000","last_clean":"2024-04-22T01:57:02.885695+0000","last_became_active":"2024-04-22T01:57:00.393053+0000","last_became_peered":"2024-04-22T01:57:00.393053+0000","last_unstale":"2024-04-22T01:57:02.885695+0000","last_undegraded":"2024-04-22T01:57:02.885695+0000","last_fullsized":"2024-04-22T01:57:02.885695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"34'237","reported_seq":289,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.310656+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:57:03.310656+0000","last_peered":"2024-04-22T01:57:03.310656+0000","last_clean":"2024-04-22T01:57:03.310656+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:57:03.310656+0000","last_undegraded":"2024-04-22T01:57:03.310656+0000","last_fullsized":"2024-04-22T01:57:03.310656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1941504,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":237,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.406614+0000","last_change":"2024-04-22T01:57:00.408300+0000","last_active":"2024-04-22T01:57:00.406614+0000","last_peered":"2024-04-22T01:57:00.406614+0000","last_clean":"2024-04-22T01:56:21.335970+0000","last_became_active":"2024-04-22T01:57:00.399660+0000","last_became_peered":"2024-04-22T01:57:00.399660+0000","last_unstale":"2024-04-22T01:57:00.406614+0000","last_undegraded":"2024-04-22T01:57:00.406614+0000","last_fullsized":"2024-04-22T01:57:00.406614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"34'232","reported_seq":283,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.555747+0000","last_change":"2024-04-22T01:57:00.384395+0000","last_active":"2024-04-22T01:57:02.555747+0000","last_peered":"2024-04-22T01:57:02.555747+0000","last_clean":"2024-04-22T01:57:02.555747+0000","last_became_active":"2024-04-22T01:57:00.373768+0000","last_became_peered":"2024-04-22T01:57:00.373768+0000","last_unstale":"2024-04-22T01:57:02.555747+0000","last_undegraded":"2024-04-22T01:57:02.555747+0000","last_fullsized":"2024-04-22T01:57:02.555747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908736,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":1864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.385411+0000","last_change":"2024-04-22T01:57:00.387131+0000","last_active":"2024-04-22T01:57:00.385411+0000","last_peered":"2024-04-22T01:57:00.385411+0000","last_clean":"2024-04-22T01:56:19.115763+0000","last_became_active":"2024-04-22T01:57:00.379810+0000","last_became_peered":"2024-04-22T01:57:00.379810+0000","last_unstale":"2024-04-22T01:57:00.385411+0000","last_undegraded":"2024-04-22T01:57:00.385411+0000","last_fullsized":"2024-04-22T01:57:00.385411+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"34'303","reported_seq":372,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:01.089526+0000","last_change":"2024-04-22T01:57:00.551107+0000","last_active":"2024-04-22T01:57:01.089526+0000","last_peered":"2024-04-22T01:57:01.089526+0000","last_clean":"2024-04-22T01:57:01.089526+0000","last_became_active":"2024-04-22T01:57:00.403784+0000","last_became_peered":"2024-04-22T01:57:00.403784+0000","last_unstale":"2024-04-22T01:57:01.089526+0000","last_undegraded":"2024-04-22T01:57:01.089526+0000","last_fullsized":"2024-04-22T01:57:01.089526+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.425482+0000","last_change":"2024-04-22T01:57:00.426750+0000","last_active":"2024-04-22T01:57:00.425482+0000","last_peered":"2024-04-22T01:57:00.425482+0000","last_clean":"2024-04-22T01:56:21.335684+0000","last_became_active":"2024-04-22T01:57:00.420756+0000","last_became_peered":"2024-04-22T01:57:00.420756+0000","last_unstale":"2024-04-22T01:57:00.425482+0000","last_undegraded":"2024-04-22T01:57:00.425482+0000","last_fullsized":"2024-04-22T01:57:00.425482+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"34'295","reported_seq":345,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.787785+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:57:02.787785+0000","last_peered":"2024-04-22T01:57:02.787785+0000","last_clean":"2024-04-22T01:57:02.787785+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:57:02.787785+0000","last_undegraded":"2024-04-22T01:57:02.787785+0000","last_fullsized":"2024-04-22T01:57:02.787785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.315306+0000","last_change":"2024-04-22T01:57:00.383516+0000","last_active":"2024-04-22T01:57:03.315306+0000","last_peered":"2024-04-22T01:57:03.315306+0000","last_clean":"2024-04-22T01:57:03.315306+0000","last_became_active":"2024-04-22T01:57:00.377612+0000","last_became_peered":"2024-04-22T01:57:00.377612+0000","last_unstale":"2024-04-22T01:57:03.315306+0000","last_undegraded":"2024-04-22T01:57:03.315306+0000","last_fullsized":"2024-04-22T01:57:03.315306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":30547968,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":29832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3719,"ondisk_log_size":3719,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52492,"kb_used_data":13188,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94319348,"statfs":{"total":96636764160,"available":96583012352,"internally_reserved":0,"allocated":13504512,"data_stored":27010716,"data_compressed":1388068,"data_compressed_allocated":13201408,"data_compressed_original":26894336,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49512,"kb_used_data":10144,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94322328,"statfs":{"total":96636764160,"available":96586063872,"internally_reserved":0,"allocated":10387456,"data_stored":20776604,"data_compressed":1066405,"data_compressed_allocated":10084352,"data_compressed_original":20660224,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50680,"kb_used_data":11312,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94321160,"statfs":{"total":96636764160,"available":96584867840,"internally_reserved":0,"allocated":11583488,"data_stored":22701180,"data_compressed":1166221,"data_compressed_allocated":11300864,"data_compressed_original":22601728,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":33,"seq":141733920772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49668,"kb_used_data":10044,"kb_used_omap":0,"kb_used_meta":39616,"kb_avail":94322172,"statfs":{"total":96636764160,"available":96585904128,"internally_reserved":0,"allocated":10285056,"data_stored":20579996,"data_compressed":1056240,"data_compressed_allocated":9981952,"data_compressed_original":20463616,"omap_allocated":0,"internal_metadata":40566784},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:57:06.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:57:06.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:06.812+0000 7fbc41a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:06.821+0000 7fbc41a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:07.051 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:57:07.052 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:57:07.068 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-22T01:57:06.283677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":31138336,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3866,"num_write_kb":31726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3816,"ondisk_log_size":3816,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":202352,"kb_used_data":44688,"kb_used_omap":0,"kb_used_meta":157632,"kb_avail":377285008,"statfs":{"total":386547056640,"available":386339848192,"internally_reserved":0,"allocated":45760512,"data_stored":91068496,"data_compressed":4676934,"data_compressed_allocated":44568576,"data_compressed_original":90619904,"omap_allocated":0,"internal_metadata":161415168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8085504,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":987,"num_write_kb":7896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082963"},"pg_stats":[{"pgid":"3.f","version":"35'215","reported_seq":280,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.368513+0000","last_change":"2024-04-22T01:57:00.436335+0000","last_active":"2024-04-22T01:57:03.368513+0000","last_peered":"2024-04-22T01:57:03.368513+0000","last_clean":"2024-04-22T01:57:03.368513+0000","last_became_active":"2024-04-22T01:57:00.426790+0000","last_became_peered":"2024-04-22T01:57:00.426790+0000","last_unstale":"2024-04-22T01:57:03.368513+0000","last_undegraded":"2024-04-22T01:57:03.368513+0000","last_fullsized":"2024-04-22T01:57:03.368513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"34'245","reported_seq":295,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.976100+0000","last_change":"2024-04-22T01:55:59.446547+0000","last_active":"2024-04-22T01:57:02.976100+0000","last_peered":"2024-04-22T01:57:02.976100+0000","last_clean":"2024-04-22T01:57:02.976100+0000","last_became_active":"2024-04-22T01:55:59.441613+0000","last_became_peered":"2024-04-22T01:55:59.441613+0000","last_unstale":"2024-04-22T01:57:02.976100+0000","last_undegraded":"2024-04-22T01:57:02.976100+0000","last_fullsized":"2024-04-22T01:57:02.976100+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2015232,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":246,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"34'216","reported_seq":266,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.315969+0000","last_change":"2024-04-22T01:57:00.396628+0000","last_active":"2024-04-22T01:57:03.315969+0000","last_peered":"2024-04-22T01:57:03.315969+0000","last_clean":"2024-04-22T01:57:03.315969+0000","last_became_active":"2024-04-22T01:57:00.392365+0000","last_became_peered":"2024-04-22T01:57:00.392365+0000","last_unstale":"2024-04-22T01:57:03.315969+0000","last_undegraded":"2024-04-22T01:57:03.315969+0000","last_fullsized":"2024-04-22T01:57:03.315969+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"35'303","reported_seq":347,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.555870+0000","last_change":"2024-04-22T01:57:00.465981+0000","last_active":"2024-04-22T01:57:03.555870+0000","last_peered":"2024-04-22T01:57:03.555870+0000","last_clean":"2024-04-22T01:57:03.555870+0000","last_became_active":"2024-04-22T01:57:00.458041+0000","last_became_peered":"2024-04-22T01:57:00.458041+0000","last_unstale":"2024-04-22T01:57:03.555870+0000","last_undegraded":"2024-04-22T01:57:03.555870+0000","last_fullsized":"2024-04-22T01:57:03.555870+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"34'199","reported_seq":259,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.922852+0000","last_change":"2024-04-22T01:57:00.553609+0000","last_active":"2024-04-22T01:57:02.922852+0000","last_peered":"2024-04-22T01:57:02.922852+0000","last_clean":"2024-04-22T01:57:02.922852+0000","last_became_active":"2024-04-22T01:57:00.404852+0000","last_became_peered":"2024-04-22T01:57:00.404852+0000","last_unstale":"2024-04-22T01:57:02.922852+0000","last_undegraded":"2024-04-22T01:57:02.922852+0000","last_fullsized":"2024-04-22T01:57:02.922852+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"31'200","reported_seq":256,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.303339+0000","last_change":"2024-04-22T01:57:00.449671+0000","last_active":"2024-04-22T01:57:03.303339+0000","last_peered":"2024-04-22T01:57:03.303339+0000","last_clean":"2024-04-22T01:57:03.303339+0000","last_became_active":"2024-04-22T01:57:00.445304+0000","last_became_peered":"2024-04-22T01:57:00.445304+0000","last_unstale":"2024-04-22T01:57:03.303339+0000","last_undegraded":"2024-04-22T01:57:03.303339+0000","last_fullsized":"2024-04-22T01:57:03.303339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"34'176","reported_seq":236,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.301038+0000","last_change":"2024-04-22T01:57:00.450117+0000","last_active":"2024-04-22T01:57:03.301038+0000","last_peered":"2024-04-22T01:57:03.301038+0000","last_clean":"2024-04-22T01:57:03.301038+0000","last_became_active":"2024-04-22T01:57:00.439954+0000","last_became_peered":"2024-04-22T01:57:00.439954+0000","last_unstale":"2024-04-22T01:57:03.301038+0000","last_undegraded":"2024-04-22T01:57:03.301038+0000","last_fullsized":"2024-04-22T01:57:03.301038+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"34'272","reported_seq":324,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.308062+0000","last_change":"2024-04-22T01:55:59.508588+0000","last_active":"2024-04-22T01:57:03.308062+0000","last_peered":"2024-04-22T01:57:03.308062+0000","last_clean":"2024-04-22T01:57:03.308062+0000","last_became_active":"2024-04-22T01:55:59.501707+0000","last_became_peered":"2024-04-22T01:55:59.501707+0000","last_unstale":"2024-04-22T01:57:03.308062+0000","last_undegraded":"2024-04-22T01:57:03.308062+0000","last_fullsized":"2024-04-22T01:57:03.308062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":208,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.363738+0000","last_change":"2024-04-22T01:57:00.364902+0000","last_active":"2024-04-22T01:57:00.363738+0000","last_peered":"2024-04-22T01:57:00.363738+0000","last_clean":"2024-04-22T01:56:19.101660+0000","last_became_active":"2024-04-22T01:57:00.359940+0000","last_became_peered":"2024-04-22T01:57:00.359940+0000","last_unstale":"2024-04-22T01:57:00.363738+0000","last_undegraded":"2024-04-22T01:57:00.363738+0000","last_fullsized":"2024-04-22T01:57:00.363738+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.312018+0000","last_change":"2024-04-22T01:57:00.404788+0000","last_active":"2024-04-22T01:57:03.312018+0000","last_peered":"2024-04-22T01:57:03.312018+0000","last_clean":"2024-04-22T01:57:03.312018+0000","last_became_active":"2024-04-22T01:57:00.399860+0000","last_became_peered":"2024-04-22T01:57:00.399860+0000","last_unstale":"2024-04-22T01:57:03.312018+0000","last_undegraded":"2024-04-22T01:57:03.312018+0000","last_fullsized":"2024-04-22T01:57:03.312018+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"34'240","reported_seq":294,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.297876+0000","last_change":"2024-04-22T01:57:00.367651+0000","last_active":"2024-04-22T01:57:03.297876+0000","last_peered":"2024-04-22T01:57:03.297876+0000","last_clean":"2024-04-22T01:57:03.297876+0000","last_became_active":"2024-04-22T01:57:00.362726+0000","last_became_peered":"2024-04-22T01:57:00.362726+0000","last_unstale":"2024-04-22T01:57:03.297876+0000","last_undegraded":"2024-04-22T01:57:03.297876+0000","last_fullsized":"2024-04-22T01:57:03.297876+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.309476+0000","last_change":"2024-04-22T01:57:00.415221+0000","last_active":"2024-04-22T01:57:03.309476+0000","last_peered":"2024-04-22T01:57:03.309476+0000","last_clean":"2024-04-22T01:57:03.309476+0000","last_became_active":"2024-04-22T01:57:00.410397+0000","last_became_peered":"2024-04-22T01:57:00.410397+0000","last_unstale":"2024-04-22T01:57:03.309476+0000","last_undegraded":"2024-04-22T01:57:03.309476+0000","last_fullsized":"2024-04-22T01:57:03.309476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"34'191","reported_seq":255,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:01.708391+0000","last_change":"2024-04-22T01:57:00.379491+0000","last_active":"2024-04-22T01:57:01.708391+0000","last_peered":"2024-04-22T01:57:01.708391+0000","last_clean":"2024-04-22T01:57:01.708391+0000","last_became_active":"2024-04-22T01:57:00.373330+0000","last_became_peered":"2024-04-22T01:57:00.373330+0000","last_unstale":"2024-04-22T01:57:01.708391+0000","last_undegraded":"2024-04-22T01:57:01.708391+0000","last_fullsized":"2024-04-22T01:57:01.708391+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.314695+0000","last_change":"2024-04-22T01:55:59.501210+0000","last_active":"2024-04-22T01:57:03.314695+0000","last_peered":"2024-04-22T01:57:03.314695+0000","last_clean":"2024-04-22T01:57:03.314695+0000","last_became_active":"2024-04-22T01:55:59.494206+0000","last_became_peered":"2024-04-22T01:55:59.494206+0000","last_unstale":"2024-04-22T01:57:03.314695+0000","last_undegraded":"2024-04-22T01:57:03.314695+0000","last_fullsized":"2024-04-22T01:57:03.314695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"34'244","reported_seq":302,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.183241+0000","last_change":"2024-04-22T01:55:59.626501+0000","last_active":"2024-04-22T01:57:03.183241+0000","last_peered":"2024-04-22T01:57:03.183241+0000","last_clean":"2024-04-22T01:57:03.183241+0000","last_became_active":"2024-04-22T01:55:59.464296+0000","last_became_peered":"2024-04-22T01:55:59.464296+0000","last_unstale":"2024-04-22T01:57:03.183241+0000","last_undegraded":"2024-04-22T01:57:03.183241+0000","last_fullsized":"2024-04-22T01:57:03.183241+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007040,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":245,"num_write_kb":1960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":65,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.303795+0000","last_change":"2024-04-22T01:57:00.468645+0000","last_active":"2024-04-22T01:57:03.303795+0000","last_peered":"2024-04-22T01:57:03.303795+0000","last_clean":"2024-04-22T01:57:03.303795+0000","last_became_active":"2024-04-22T01:57:00.462213+0000","last_became_peered":"2024-04-22T01:57:00.462213+0000","last_unstale":"2024-04-22T01:57:03.303795+0000","last_undegraded":"2024-04-22T01:57:03.303795+0000","last_fullsized":"2024-04-22T01:57:03.303795+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"34'151","reported_seq":207,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.885695+0000","last_change":"2024-04-22T01:57:00.571514+0000","last_active":"2024-04-22T01:57:02.885695+0000","last_peered":"2024-04-22T01:57:02.885695+0000","last_clean":"2024-04-22T01:57:02.885695+0000","last_became_active":"2024-04-22T01:57:00.393053+0000","last_became_peered":"2024-04-22T01:57:00.393053+0000","last_unstale":"2024-04-22T01:57:02.885695+0000","last_undegraded":"2024-04-22T01:57:02.885695+0000","last_fullsized":"2024-04-22T01:57:02.885695+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"34'237","reported_seq":289,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.310656+0000","last_change":"2024-04-22T01:55:59.551363+0000","last_active":"2024-04-22T01:57:03.310656+0000","last_peered":"2024-04-22T01:57:03.310656+0000","last_clean":"2024-04-22T01:57:03.310656+0000","last_became_active":"2024-04-22T01:55:59.539360+0000","last_became_peered":"2024-04-22T01:55:59.539360+0000","last_unstale":"2024-04-22T01:57:03.310656+0000","last_undegraded":"2024-04-22T01:57:03.310656+0000","last_fullsized":"2024-04-22T01:57:03.310656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1941504,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":237,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.406614+0000","last_change":"2024-04-22T01:57:00.408300+0000","last_active":"2024-04-22T01:57:00.406614+0000","last_peered":"2024-04-22T01:57:00.406614+0000","last_clean":"2024-04-22T01:56:21.335970+0000","last_became_active":"2024-04-22T01:57:00.399660+0000","last_became_peered":"2024-04-22T01:57:00.399660+0000","last_unstale":"2024-04-22T01:57:00.406614+0000","last_undegraded":"2024-04-22T01:57:00.406614+0000","last_fullsized":"2024-04-22T01:57:00.406614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"34'232","reported_seq":283,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.555747+0000","last_change":"2024-04-22T01:57:00.384395+0000","last_active":"2024-04-22T01:57:02.555747+0000","last_peered":"2024-04-22T01:57:02.555747+0000","last_clean":"2024-04-22T01:57:02.555747+0000","last_became_active":"2024-04-22T01:57:00.373768+0000","last_became_peered":"2024-04-22T01:57:00.373768+0000","last_unstale":"2024-04-22T01:57:02.555747+0000","last_undegraded":"2024-04-22T01:57:02.555747+0000","last_fullsized":"2024-04-22T01:57:02.555747+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908736,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":233,"num_write_kb":1864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.385411+0000","last_change":"2024-04-22T01:57:00.387131+0000","last_active":"2024-04-22T01:57:00.385411+0000","last_peered":"2024-04-22T01:57:00.385411+0000","last_clean":"2024-04-22T01:56:19.115763+0000","last_became_active":"2024-04-22T01:57:00.379810+0000","last_became_peered":"2024-04-22T01:57:00.379810+0000","last_unstale":"2024-04-22T01:57:00.385411+0000","last_undegraded":"2024-04-22T01:57:00.385411+0000","last_fullsized":"2024-04-22T01:57:00.385411+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"34'303","reported_seq":372,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:01.089526+0000","last_change":"2024-04-22T01:57:00.551107+0000","last_active":"2024-04-22T01:57:01.089526+0000","last_peered":"2024-04-22T01:57:01.089526+0000","last_clean":"2024-04-22T01:57:01.089526+0000","last_became_active":"2024-04-22T01:57:00.403784+0000","last_became_peered":"2024-04-22T01:57:00.403784+0000","last_unstale":"2024-04-22T01:57:01.089526+0000","last_undegraded":"2024-04-22T01:57:01.089526+0000","last_fullsized":"2024-04-22T01:57:01.089526+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:00.425482+0000","last_change":"2024-04-22T01:57:00.426750+0000","last_active":"2024-04-22T01:57:00.425482+0000","last_peered":"2024-04-22T01:57:00.425482+0000","last_clean":"2024-04-22T01:56:21.335684+0000","last_became_active":"2024-04-22T01:57:00.420756+0000","last_became_peered":"2024-04-22T01:57:00.420756+0000","last_unstale":"2024-04-22T01:57:00.425482+0000","last_undegraded":"2024-04-22T01:57:00.425482+0000","last_fullsized":"2024-04-22T01:57:00.425482+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"34'295","reported_seq":345,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-22T01:57:02.787785+0000","last_change":"2024-04-22T01:55:59.451473+0000","last_active":"2024-04-22T01:57:02.787785+0000","last_peered":"2024-04-22T01:57:02.787785+0000","last_clean":"2024-04-22T01:57:02.787785+0000","last_became_active":"2024-04-22T01:55:59.445608+0000","last_became_peered":"2024-04-22T01:55:59.445608+0000","last_unstale":"2024-04-22T01:57:02.787785+0000","last_undegraded":"2024-04-22T01:57:02.787785+0000","last_fullsized":"2024-04-22T01:57:02.787785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:35.287370+0000","last_clean_scrub_stamp":"2024-04-22T01:55:35.287370+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-22T01:57:03.315306+0000","last_change":"2024-04-22T01:57:00.383516+0000","last_active":"2024-04-22T01:57:03.315306+0000","last_peered":"2024-04-22T01:57:03.315306+0000","last_clean":"2024-04-22T01:57:03.315306+0000","last_became_active":"2024-04-22T01:57:00.377612+0000","last_became_peered":"2024-04-22T01:57:00.377612+0000","last_unstale":"2024-04-22T01:57:03.315306+0000","last_undegraded":"2024-04-22T01:57:03.315306+0000","last_fullsized":"2024-04-22T01:57:03.315306+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":30547968,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3729,"num_write_kb":29832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":57344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3719,"ondisk_log_size":3719,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52492,"kb_used_data":13188,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94319348,"statfs":{"total":96636764160,"available":96583012352,"internally_reserved":0,"allocated":13504512,"data_stored":27010716,"data_compressed":1388068,"data_compressed_allocated":13201408,"data_compressed_original":26894336,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49512,"kb_used_data":10144,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94322328,"statfs":{"total":96636764160,"available":96586063872,"internally_reserved":0,"allocated":10387456,"data_stored":20776604,"data_compressed":1066405,"data_compressed_allocated":10084352,"data_compressed_original":20660224,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":24,"seq":103079215168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50680,"kb_used_data":11312,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":94321160,"statfs":{"total":96636764160,"available":96584867840,"internally_reserved":0,"allocated":11583488,"data_stored":22701180,"data_compressed":1166221,"data_compressed_allocated":11300864,"data_compressed_original":22601728,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":33,"seq":141733920772,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49668,"kb_used_data":10044,"kb_used_omap":0,"kb_used_meta":39616,"kb_avail":94322172,"statfs":{"total":96636764160,"available":96585904128,"internally_reserved":0,"allocated":10285056,"data_stored":20579996,"data_compressed":1056240,"data_compressed_allocated":9981952,"data_compressed_original":20463616,"omap_allocated":0,"internal_metadata":40566784},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:57:07.069 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T01:57:07.069 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:07.069 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [3, 2, 1, 0] 2024-04-22T01:57:07.070 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T01:57:07.070 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:57:07.147 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T01:57:07.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 16 4314 4298 0.373049 0.890625 0.081789 0.333691 2024-04-22T01:57:07.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:08.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:08.010+0000 7f5b35f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:08.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:08.015+0000 7f5b35f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:08.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 91 16 4314 4298 0.36895 0 - 0.333691 2024-04-22T01:57:09.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 92 16 4314 4298 0.364939 0 - 0.333691 2024-04-22T01:57:10.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 93 16 4314 4298 0.361015 0 - 0.333691 2024-04-22T01:57:11.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 94 16 4314 4298 0.357174 0 - 0.333691 2024-04-22T01:57:12.148 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [3, 2, 0] 2024-04-22T01:57:12.148 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:12.148 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-22T01:57:12.148 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T01:57:12.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T01:57:12.151 INFO:tasks.ceph.osd.1:Started 2024-04-22T01:57:12.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:57:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:57:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:12.265 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T01:57:12.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 95 16 4314 4298 0.353415 0 - 0.333691 2024-04-22T01:57:13.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 96 16 4314 4298 0.349733 0 - 0.333691 2024-04-22T01:57:14.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 97 16 4314 4298 0.346128 0 - 0.333691 2024-04-22T01:57:15.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 98 16 4314 4298 0.342596 0 - 0.333691 2024-04-22T01:57:16.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 99 16 4314 4298 0.339135 0 - 0.333691 2024-04-22T01:57:17.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:57:17.365 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:57:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:17.368 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T01:57:17.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:57:17.398796+0000 min lat: 0.0159625 max lat: 16.5699 avg lat: 0.333691 2024-04-22T01:57:17.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:57:17.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 100 16 4314 4298 0.335744 0 - 0.333691 2024-04-22T01:57:18.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 101 16 4314 4298 0.33242 0 - 0.333691 2024-04-22T01:57:19.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 102 16 4314 4298 0.329161 0 - 0.333691 2024-04-22T01:57:20.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 103 16 4314 4298 0.325965 0 - 0.333691 2024-04-22T01:57:21.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 104 16 4314 4298 0.322831 0 - 0.333691 2024-04-22T01:57:22.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:57:22.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 105 16 4314 4298 0.319757 0 - 0.333691 2024-04-22T01:57:22.472 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:57:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:22.475 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-22T01:57:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:57:23.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 106 16 4314 4298 0.31674 0 - 0.333691 2024-04-22T01:57:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:57:23.470 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:57:24.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 107 16 4314 4298 0.31378 0 - 0.333691 2024-04-22T01:57:25.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 108 16 4314 4298 0.310875 0 - 0.333691 2024-04-22T01:57:25.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:25.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:25.608+0000 7fe4498ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:25.617 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:25.618+0000 7fe4498ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 108.295 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 4314 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 8192 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 0.311216 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 0.521789 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 1.42188 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 39 2024-04-22T01:57:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 66.7939 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 182 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.401432 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 2.08535 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 18.6823 2024-04-22T01:57:26.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.0159625 2024-04-22T01:57:26.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-04-22T01:57:27.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-22T01:57:27.653 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:57:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [ 2024-04-22T01:57:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "replicated_request", 2024-04-22T01:57:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 388, 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "RepRequest": { 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "reqid": "client.4233.0:4565", 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "3.a", 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 41, 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "min_epoch": 40, 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "oid": "3:567b7815:::benchmark_data_smithi107_29052_object248:head", 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "0" 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 194, 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.422581+0000" 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.422592+0000" 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.422750+0000" 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.422857+0000" 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.422952+0000" 2024-04-22T01:57:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.481462+0000" 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 205, 2024-04-22T01:57:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.482306+0000" 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.482322+0000" 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.484050+0000" 2024-04-22T01:57:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.484256+0000" 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.484438+0000" 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.540312+0000" 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 213, 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.531093+0000" 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.531109+0000" 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.533329+0000" 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.533472+0000" 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.534928+0000" 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.593241+0000" 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 217, 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.554214+0000" 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.554237+0000" 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.554745+0000" 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.555027+0000" 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.555387+0000" 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.658037+0000" 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 225, 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.605011+0000" 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.605039+0000" 2024-04-22T01:57:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.605535+0000" 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.608393+0000" 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.609520+0000" 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.712550+0000" 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 239, 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.699093+0000" 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.699106+0000" 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.699376+0000" 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.701219+0000" 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.701443+0000" 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.829019+0000" 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 244, 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.731107+0000" 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.731123+0000" 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.731508+0000" 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.731825+0000" 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.732443+0000" 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.837805+0000" 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 247, 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.745887+0000" 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.745897+0000" 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.749586+0000" 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.750194+0000" 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.751151+0000" 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.849930+0000" 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 250, 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.777495+0000" 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.777515+0000" 2024-04-22T01:57:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.780486+0000" 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.780968+0000" 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.783056+0000" 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.865784+0000" 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 260, 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.851022+0000" 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.851030+0000" 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.851233+0000" 2024-04-22T01:57:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.851330+0000" 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.851423+0000" 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.936338+0000" 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 282, 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.969853+0000" 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.969862+0000" 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.970069+0000" 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.970193+0000" 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.970373+0000" 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.065489+0000" 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 291, 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.019849+0000" 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.019863+0000" 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.020511+0000" 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.020903+0000" 2024-04-22T01:57:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.021567+0000" 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.098581+0000" 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 300, 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.078500+0000" 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.078516+0000" 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.078662+0000" 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.078807+0000" 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.079386+0000" 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.202159+0000" 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 323, 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.208872+0000" 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.208887+0000" 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.209109+0000" 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.209268+0000" 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.209964+0000" 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.301734+0000" 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 330, 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.263969+0000" 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.263987+0000" 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.264147+0000" 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.264368+0000" 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.264544+0000" 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.317519+0000" 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 346, 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.343641+0000" 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.343655+0000" 2024-04-22T01:57:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.345674+0000" 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.345822+0000" 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.347092+0000" 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.433450+0000" 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 350, 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.378556+0000" 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.378571+0000" 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.378848+0000" 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.378987+0000" 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.379357+0000" 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.440986+0000" 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 379, 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.552700+0000" 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.552713+0000" 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.553222+0000" 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.553686+0000" 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.554115+0000" 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.611507+0000" 2024-04-22T01:57:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "replicated_request", 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 385, 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "RepRequest": { 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "reqid": "client.4233.0:4562", 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "3.7", 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 41, 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "min_epoch": 40, 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "oid": "3:e5caac34:::benchmark_data_smithi107_29052_object245:head", 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "3" 2024-04-22T01:57:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 196, 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.425787+0000" 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.425795+0000" 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.425986+0000" 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.426332+0000" 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.430963+0000" 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.492629+0000" 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 197, 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.431434+0000" 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.431455+0000" 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.431764+0000" 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.432991+0000" 2024-04-22T01:57:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.434579+0000" 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.516872+0000" 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 208, 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.501644+0000" 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.501673+0000" 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.502071+0000" 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.504656+0000" 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.507736+0000" 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.585330+0000" 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 209, 2024-04-22T01:57:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.509465+0000" 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.509477+0000" 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.509765+0000" 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.509902+0000" 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.517160+0000" 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.586790+0000" 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 242, 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.727039+0000" 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.727054+0000" 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.727231+0000" 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.727428+0000" 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.728347+0000" 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.839707+0000" 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 269, 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.894863+0000" 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.894876+0000" 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.720 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.897117+0000" 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.900388+0000" 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.901156+0000" 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.983983+0000" 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.722 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 281, 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.961784+0000" 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.961799+0000" 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.962194+0000" 2024-04-22T01:57:27.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.963333+0000" 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.965536+0000" 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.724 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.057801+0000" 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 292, 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.025559+0000" 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.025569+0000" 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.025919+0000" 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.726 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.027052+0000" 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.027426+0000" 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.727 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.085346+0000" 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 295, 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.728 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.039488+0000" 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.039504+0000" 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.041617+0000" 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.729 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.044611+0000" 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.047501+0000" 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.120499+0000" 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 298, 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.066474+0000" 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.066485+0000" 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.066628+0000" 2024-04-22T01:57:27.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.066839+0000" 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.068819+0000" 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.732 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.188743+0000" 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 302, 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.733 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.087011+0000" 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.087032+0000" 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.087396+0000" 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.734 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.089711+0000" 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.093248+0000" 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204033+0000" 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 318, 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.736 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.171789+0000" 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.171804+0000" 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.173335+0000" 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.737 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.175543+0000" 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.176853+0000" 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.258841+0000" 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 321, 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204039+0000" 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.739 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204049+0000" 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204211+0000" 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.740 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204499+0000" 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204918+0000" 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.267842+0000" 2024-04-22T01:57:27.741 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 332, 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.272170+0000" 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.742 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.272186+0000" 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.275500+0000" 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.277932+0000" 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.743 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.281355+0000" 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.380559+0000" 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 335, 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.286843+0000" 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.286857+0000" 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.287067+0000" 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.287236+0000" 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.746 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.293375+0000" 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.393104+0000" 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 337, 2024-04-22T01:57:27.747 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.304382+0000" 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.304403+0000" 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.305311+0000" 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.305513+0000" 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.308387+0000" 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.398829+0000" 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 342, 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.750 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.329484+0000" 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.329500+0000" 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.329866+0000" 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.751 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.330059+0000" 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.333455+0000" 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.752 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.421920+0000" 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 360, 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.447703+0000" 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.753 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.447727+0000" 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.448061+0000" 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.448263+0000" 2024-04-22T01:57:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.449370+0000" 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.536169+0000" 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 362, 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.451789+0000" 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.451807+0000" 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.756 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.452069+0000" 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.452233+0000" 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.453682+0000" 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.544357+0000" 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.758 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 369, 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.491253+0000" 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.491269+0000" 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.492701+0000" 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.759 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.492909+0000" 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.493221+0000" 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.558922+0000" 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 380, 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.761 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.557783+0000" 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.557801+0000" 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.557929+0000" 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.762 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.558046+0000" 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.563952+0000" 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.637107+0000" 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 61, 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.436248+0000" 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.436273+0000" 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.436944+0000" 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.437820+0000" 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.765 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.438584+0000" 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.588968+0000" 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.766 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 63, 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.440301+0000" 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.440310+0000" 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.440446+0000" 2024-04-22T01:57:27.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.440822+0000" 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.444704+0000" 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.768 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.611038+0000" 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 77, 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.458352+0000" 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.458363+0000" 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.458859+0000" 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.459007+0000" 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.459792+0000" 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.771 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.631708+0000" 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 67, 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.448060+0000" 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.772 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.448075+0000" 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.448692+0000" 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.449314+0000" 2024-04-22T01:57:27.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.450736+0000" 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.632301+0000" 2024-04-22T01:57:27.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 79, 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.460369+0000" 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.775 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.460377+0000" 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.460713+0000" 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.461431+0000" 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.776 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.462576+0000" 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.657179+0000" 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.777 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 72, 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.452243+0000" 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.452256+0000" 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.452537+0000" 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.778 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.453214+0000" 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.454806+0000" 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.779 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.657682+0000" 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 74, 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.455978+0000" 2024-04-22T01:57:27.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.455992+0000" 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.456167+0000" 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.456781+0000" 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:24.457469+0000" 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:25.682259+0000" 2024-04-22T01:57:27.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 195, 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.424356+0000" 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.424367+0000" 2024-04-22T01:57:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.424610+0000" 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.424739+0000" 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.426646+0000" 2024-04-22T01:57:27.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.482648+0000" 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 199, 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.785 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.436484+0000" 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.436499+0000" 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.436891+0000" 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.439211+0000" 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.441423+0000" 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.518227+0000" 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 206, 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.490127+0000" 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.788 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.490137+0000" 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.490995+0000" 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.491597+0000" 2024-04-22T01:57:27.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.493187+0000" 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.573559+0000" 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 211, 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.511079+0000" 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.511090+0000" 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.791 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.512125+0000" 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.514022+0000" 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.514964+0000" 2024-04-22T01:57:27.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.587166+0000" 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 218, 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.556131+0000" 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.556141+0000" 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.556467+0000" 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.556832+0000" 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.558959+0000" 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.629813+0000" 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 220, 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.584093+0000" 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.796 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.584109+0000" 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.584401+0000" 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.584549+0000" 2024-04-22T01:57:27.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.584821+0000" 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.667296+0000" 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 222, 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.589239+0000" 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.589253+0000" 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.799 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.589801+0000" 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.590135+0000" 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.591806+0000" 2024-04-22T01:57:27.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.688544+0000" 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 229, 2024-04-22T01:57:27.801 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.641434+0000" 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.641449+0000" 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.642071+0000" 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.802 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.642923+0000" 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.643610+0000" 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.727705+0000" 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 234, 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.804 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.675653+0000" 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.675680+0000" 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.676426+0000" 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.676684+0000" 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.679898+0000" 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.806 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.790466+0000" 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 248, 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.760519+0000" 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.807 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.760528+0000" 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.761150+0000" 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.808 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.761412+0000" 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.765674+0000" 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.842618+0000" 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 254, 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.809738+0000" 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.810 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.809760+0000" 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.809984+0000" 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.810356+0000" 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.811 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.813054+0000" 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.877105+0000" 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 262, 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.855507+0000" 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.855516+0000" 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.813 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.856050+0000" 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.859307+0000" 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.814 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.859893+0000" 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.941940+0000" 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.815 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 267, 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.891396+0000" 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.891416+0000" 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.891631+0000" 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.816 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.892088+0000" 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.892533+0000" 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.817 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.964494+0000" 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 277, 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.818 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.938567+0000" 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.938601+0000" 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.938848+0000" 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.939108+0000" 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.940344+0000" 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.048185+0000" 2024-04-22T01:57:27.820 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 284, 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.974915+0000" 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.974930+0000" 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.821 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.975235+0000" 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.977059+0000" 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.822 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.980421+0000" 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.067616+0000" 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 285, 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.982585+0000" 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.982600+0000" 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.985616+0000" 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.824 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.988805+0000" 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:26.994348+0000" 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.068359+0000" 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 287, 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.000178+0000" 2024-04-22T01:57:27.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.000202+0000" 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.000430+0000" 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.827 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.000740+0000" 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.002303+0000" 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.069944+0000" 2024-04-22T01:57:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 289, 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.004022+0000" 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.004039+0000" 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.829 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.006833+0000" 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.011478+0000" 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.012985+0000" 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.830 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.070683+0000" 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 303, 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.831 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.098708+0000" 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.098721+0000" 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.101088+0000" 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.832 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.102459+0000" 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.103614+0000" 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.833 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.204607+0000" 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 309, 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.134031+0000" 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.834 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.134045+0000" 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.134447+0000" 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.835 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.136821+0000" 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.137490+0000" 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.206640+0000" 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 311, 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.138637+0000" 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.138647+0000" 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.138873+0000" 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.139251+0000" 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.838 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.140963+0000" 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.208622+0000" 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 314, 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.160684+0000" 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.160702+0000" 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.163514+0000" 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.840 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.163684+0000" 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.164081+0000" 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.841 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.213679+0000" 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 316, 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.169014+0000" 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.842 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.169030+0000" 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.169406+0000" 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.169732+0000" 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.843 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.170727+0000" 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.228294+0000" 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.844 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 334, 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.283869+0000" 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.283881+0000" 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.845 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.285558+0000" 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.285936+0000" 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.286348+0000" 2024-04-22T01:57:27.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.382563+0000" 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 339, 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.314325+0000" 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.314336+0000" 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.314547+0000" 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.848 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.314667+0000" 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.315504+0000" 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.415483+0000" 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 344, 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.850 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.340569+0000" 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.340585+0000" 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.340801+0000" 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.851 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.341088+0000" 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.341838+0000" 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.421648+0000" 2024-04-22T01:57:27.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 347, 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.354745+0000" 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.354766+0000" 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.853 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.354932+0000" 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.355068+0000" 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.854 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.356998+0000" 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.434898+0000" 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.855 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 351, 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.387529+0000" 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.387538+0000" 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.388190+0000" 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.856 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.388747+0000" 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.392752+0000" 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.442811+0000" 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 352, 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.396509+0000" 2024-04-22T01:57:27.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.396522+0000" 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.396731+0000" 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.859 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.396867+0000" 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.399081+0000" 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.445685+0000" 2024-04-22T01:57:27.860 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 363, 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.452815+0000" 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.452826+0000" 2024-04-22T01:57:27.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.453047+0000" 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.453176+0000" 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.862 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.453481+0000" 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.509275+0000" 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 370, 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.506210+0000" 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.506229+0000" 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.864 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.506488+0000" 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.506686+0000" 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.508498+0000" 2024-04-22T01:57:27.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.573630+0000" 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 374, 2024-04-22T01:57:27.866 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.521993+0000" 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.522019+0000" 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.522416+0000" 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.526528+0000" 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.529370+0000" 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.591649+0000" 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 377, 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.551100+0000" 2024-04-22T01:57:27.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.551118+0000" 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.551320+0000" 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.870 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.551527+0000" 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.552020+0000" 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:57:27.871 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:57:27.616564+0000" 2024-04-22T01:57:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:57:27.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:57:27.872 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:57:27.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:28.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:28.059+0000 7f13300ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:28.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:28.067+0000 7f13300ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:29.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 540 objects 2024-04-22T01:57:29.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :2.72242 2024-04-22T01:57:30.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:30.676+0000 7f8aa48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:30.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:30.682+0000 7f8aa48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:32.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:32.874 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 0, 1] 2024-04-22T01:57:32.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:32.875 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:57:32.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:57:32.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:32.901+0000 7f42df824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:32.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:32.907+0000 7f42df824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:32.993+0000 7ff6f482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.001+0000 7ff6f482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.016 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:33.016 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:33.016 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:33.016 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:33.017 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:33.019 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:33.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:57:33.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.164+0000 7f5685406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.173+0000 7f5685406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:33.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:33.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:57:33.252 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-22T01:57:33.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-22T01:57:33.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.296+0000 7f4c4c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.305+0000 7f4c4c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:33.321 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:33.322 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:33.322 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:33.322 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:33.324 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:33.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:57:33.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.360+0000 7fc10cb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.373+0000 7fc10cb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.448+0000 7f5ad2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:33.459+0000 7f5ad2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:33.487 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:33.490 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_0' removed 2024-04-22T01:57:34.048 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-22T01:57:34.048 INFO:tasks.radosbench:Beginning radosbench... 2024-04-22T01:57:34.048 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-22T01:57:34.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2024-04-22T01:57:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:34.192+0000 7fe9b666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:34.210+0000 7fe9b666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:35.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_1' created 2024-04-22T01:57:35.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-22T01:57:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:35.164+0000 7fe74840f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:35.171+0000 7fe74840f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:35.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:35.233+0000 7efdbc39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:35.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:35.247+0000 7efdbc39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:36.040 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-22T01:57:37.065 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-22T01:57:37.068 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-22T01:57:37.068 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-22T01:57:37.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-04-22T01:57:37.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-22T01:57:37.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_32846 2024-04-22T01:57:37.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:57:37.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-04-22T01:57:37.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:37.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:37.440+0000 7f8dc846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:37.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:37.446+0000 7f8dc846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:38.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 154 138 1.07802 1.07812 0.104007 0.108346 2024-04-22T01:57:38.492 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 0, 1] 2024-04-22T01:57:38.492 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:38.492 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:57:38.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:57:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:38.635+0000 7fa4294ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:38.647+0000 7fa4294ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:38.681 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:38.681 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:38.682 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:38.685 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:38.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:57:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:38.814+0000 7fa1a50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:38.826+0000 7fa1a50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:38.874 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:38.875 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:38.875 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:38.875 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:38.875 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:38.877 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:38.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:57:39.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.008+0000 7f608e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:39.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.019+0000 7f608e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:39.043 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:39.046 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:39.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:57:39.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 314 298 1.16395 1.25 0.0438954 0.104338 2024-04-22T01:57:39.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.175+0000 7f72c8d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:39.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.186+0000 7f72c8d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:57:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:57:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:57:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:57:39.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:57:39.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:39.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.772+0000 7fdd06a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:39.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:39.778+0000 7fdd06a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:40.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 476 460 1.1978 1.26562 0.0878798 0.102694 2024-04-22T01:57:41.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 641 625 1.22059 1.28906 0.119655 0.101053 2024-04-22T01:57:41.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:57:42.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:42.061+0000 7f450df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:42.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:42.069+0000 7f450df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:42.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 803 787 1.22958 1.26562 0.165029 0.0996769 2024-04-22T01:57:43.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 964 948 1.23426 1.25781 0.115028 0.0999672 2024-04-22T01:57:44.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 1122 1106 1.23426 1.23438 0.0700744 0.100457 2024-04-22T01:57:44.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 2, 0, 1] 2024-04-22T01:57:44.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:44.210 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [3, 2, 0, 1] 2024-04-22T01:57:44.211 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T01:57:44.211 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:57:44.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:57:44.300 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T01:57:44.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:44.363+0000 7f6ced954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:44.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:44.371+0000 7f6ced954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:45.133 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 1155 1139 1.1122 0.257812 0.123516 0.100843 2024-04-22T01:57:46.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 1155 1139 0.98862 0 - 0.100843 2024-04-22T01:57:47.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 1155 1139 0.889753 0 - 0.100843 2024-04-22T01:57:48.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 1155 1139 0.808865 0 - 0.100843 2024-04-22T01:57:49.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 1155 1139 0.741459 0 - 0.100843 2024-04-22T01:57:49.300 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 0, 1] 2024-04-22T01:57:49.301 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:49.301 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [3, 0, 1] 2024-04-22T01:57:49.302 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:57:49.302 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:57:49.385 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:57:50.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 1155 1139 0.684424 0 - 0.100843 2024-04-22T01:57:51.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 1155 1139 0.635535 0 - 0.100843 2024-04-22T01:57:52.134 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 1155 1139 0.593165 0 - 0.100843 2024-04-22T01:57:53.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 1155 1139 0.556091 0 - 0.100843 2024-04-22T01:57:54.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 1155 1139 0.523378 0 - 0.100843 2024-04-22T01:57:54.387 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 0] live_osds: [3, 1] 2024-04-22T01:57:54.387 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:57:54.387 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:57:54.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:57:54.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:54.504+0000 7ff906b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:54.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:57:54.513+0000 7ff906b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:55.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 1155 1139 0.494299 0 - 0.100843 2024-04-22T01:57:56.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 1155 1139 0.468283 0 - 0.100843 2024-04-22T01:57:57.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:57:57.137191+0000 min lat: 0.0156288 max lat: 0.296237 avg lat: 0.100843 2024-04-22T01:57:57.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:57:57.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 1155 1139 0.444869 0 - 0.100843 2024-04-22T01:57:58.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 1155 1139 0.423684 0 - 0.100843 2024-04-22T01:57:59.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 1155 1139 0.404426 0 - 0.100843 2024-04-22T01:58:00.135 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 1155 1139 0.386841 0 - 0.100843 2024-04-22T01:58:01.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 1155 1139 0.370723 0 - 0.100843 2024-04-22T01:58:02.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 1155 1139 0.355894 0 - 0.100843 2024-04-22T01:58:03.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 1155 1139 0.342205 0 - 0.100843 2024-04-22T01:58:04.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 1155 1139 0.329531 0 - 0.100843 2024-04-22T01:58:05.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 1155 1139 0.317761 0 - 0.100843 2024-04-22T01:58:06.136 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 1155 1139 0.306804 0 - 0.100843 2024-04-22T01:58:07.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 1155 1139 0.296577 0 - 0.100843 2024-04-22T01:58:08.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 1155 1139 0.28701 0 - 0.100843 2024-04-22T01:58:09.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 1155 1139 0.278041 0 - 0.100843 2024-04-22T01:58:10.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 1155 1139 0.269616 0 - 0.100843 2024-04-22T01:58:11.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 1155 1139 0.261686 0 - 0.100843 2024-04-22T01:58:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:12.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 1155 1139 0.254209 0 - 0.100843 2024-04-22T01:58:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:12.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:13.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 1156 1140 0.247365 0.000279018 28.6873 0.125919 2024-04-22T01:58:14.137 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 1180 1164 0.245747 0.1875 0.12338 0.396679 2024-04-22T01:58:14.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:14.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:14.258+0000 7f8ce4664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:14.263+0000 7f8ce4664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:15.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 1180 1164 0.239279 0 - 0.396679 2024-04-22T01:58:16.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 1180 1164 0.233144 0 - 0.396679 2024-04-22T01:58:17.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:58:17.139897+0000 min lat: 0.0156288 max lat: 29.1248 avg lat: 0.396679 2024-04-22T01:58:17.224 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:58:17.224 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 1180 1164 0.227315 0 - 0.396679 2024-04-22T01:58:18.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 1180 1164 0.221771 0 - 0.396679 2024-04-22T01:58:19.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 1180 1164 0.216491 0 - 0.396679 2024-04-22T01:58:19.266 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:19.312 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:19.520 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:19.524 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:19.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:58:19.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.627+0000 7f6f32824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.636+0000 7f6f32824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.649 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:58:19.649 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:58:19.650 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:58:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:19.653 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:19.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:58:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.756+0000 7f0c5ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.764+0000 7f0c5ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.768 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:19.771 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:19.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:58:19.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.875+0000 7fb99c39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:19.883+0000 7fb99c39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:58:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:58:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:58:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:58:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:58:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:58:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:19.902 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:20.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 1189 1173 0.213091 0.0117188 5.98953 0.46428 2024-04-22T01:58:21.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 1189 1173 0.208248 0 - 0.46428 2024-04-22T01:58:21.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:21.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:21.430+0000 7f6af466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:21.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:21.435+0000 7f6af466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:21.439 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:21.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:22.138 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 1189 1173 0.20362 0 - 0.46428 2024-04-22T01:58:23.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 1189 1173 0.199194 0 - 0.46428 2024-04-22T01:58:23.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:23.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:23.601+0000 7febdd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:23.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:23.606+0000 7febdd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:23.610 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:23.628 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:23.656 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:24.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 1189 1173 0.194955 0 - 0.46428 2024-04-22T01:58:24.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 0] live_osds: [3, 1] 2024-04-22T01:58:24.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:58:24.903 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-22T01:58:24.904 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:58:24.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:58:24.907 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:58:24.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:58:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:58:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:25.020 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:58:25.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 1189 1173 0.190894 0 - 0.46428 2024-04-22T01:58:25.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:25.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:25.761+0000 7f9031c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:25.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:25.766+0000 7f9031c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:25.770 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:25.788 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:25.814 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:26.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 1189 1173 0.186998 0 - 0.46428 2024-04-22T01:58:27.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 1189 1173 0.183258 0 - 0.46428 2024-04-22T01:58:27.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:27.920 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:27.919+0000 7f95fb2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:27.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:27.924+0000 7f95fb2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:27.928 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-22T01:58:27.947 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:27.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:28.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 1189 1173 0.179665 0 - 0.46428 2024-04-22T01:58:29.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 1189 1173 0.17621 0 - 0.46428 2024-04-22T01:58:29.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:30.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:58:30.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:30.088+0000 7f78fc99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:30.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:30.093+0000 7f78fc99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:30.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 1189 1173 0.172886 0 - 0.46428 2024-04-22T01:58:30.167 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [ 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 53, 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.387047+0000" 2024-04-22T01:58:30.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.387079+0000" 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.388268+0000" 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.388613+0000" 2024-04-22T01:58:30.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.399948+0000" 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.400331+0000" 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.170 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 55, 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.394877+0000" 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.394892+0000" 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.396318+0000" 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.172 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.400479+0000" 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.402317+0000" 2024-04-22T01:58:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 58, 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.401646+0000" 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.401655+0000" 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.175 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.402631+0000" 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.402880+0000" 2024-04-22T01:58:30.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.407075+0000" 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.177 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 61, 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.178 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.410747+0000" 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.410764+0000" 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.410916+0000" 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.179 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.411145+0000" 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.414457+0000" 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 64, 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.181 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.420481+0000" 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.420504+0000" 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.421693+0000" 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.182 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.422351+0000" 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.424173+0000" 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 68, 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.429116+0000" 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.429146+0000" 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.429986+0000" 2024-04-22T01:58:30.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.430296+0000" 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.431501+0000" 2024-04-22T01:58:30.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.187 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 71, 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.433805+0000" 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.433817+0000" 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.188 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.434087+0000" 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.434371+0000" 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.435817+0000", 2024-04-22T01:58:30.189 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.190 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 74, 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.191 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.438522+0000" 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.438535+0000" 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.439383+0000" 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.192 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.439649+0000" 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.193 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.442040+0000", 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 78, 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.444021+0000" 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.444033+0000" 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.444764+0000" 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.196 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.445156+0000" 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.445528+0000", 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.197 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 82, 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.450879+0000" 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.199 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.450894+0000" 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.451101+0000" 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.452443+0000" 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.200 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.453237+0000", 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 84, 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.453960+0000" 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.453969+0000" 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.203 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.454126+0000" 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.454377+0000" 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.204 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.454674+0000", 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.205 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 85, 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.454954+0000" 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.206 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.454962+0000" 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.455077+0000" 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.207 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.455782+0000" 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.456962+0000", 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.208 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.209 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 88, 2024-04-22T01:58:30.210 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.457449+0000" 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.457457+0000" 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.457632+0000" 2024-04-22T01:58:30.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.458190+0000" 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.212 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.459662+0000", 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 93, 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.214 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.461663+0000" 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.461678+0000" 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.461951+0000" 2024-04-22T01:58:30.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.216 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.462101+0000" 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.462486+0000", 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.217 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.219 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.220 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 94, 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.462711+0000" 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.221 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.462718+0000" 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.462930+0000" 2024-04-22T01:58:30.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.463041+0000" 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.463447+0000", 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.225 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client_request", 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 95, 2024-04-22T01:58:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.463595+0000" 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.463602+0000" 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.463794+0000" 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.464350+0000" 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:29.465275+0000", 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "blocker": { 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "op_type": "PerShardPipeline::create_or_wait_pg", 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "waiting": 10, 2024-04-22T01:58:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: "held_by_operation_id": 68 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "blocking_event": { 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time_event": { 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "peering_event", 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 111, 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "PeeringEvent": { 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "3", 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.c", 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "sent": 51, 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "requested": 51, 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "evt": "epoch_sent: 51 epoch_requested: 51 MInfoRec from 3 info: 4.c( v 45'72 (0'0,45'72] local-lis/les=50/51 n=9 ec=43/43 lis/c=48/43 les/c/f=49/44/0 sis=50) pg_lease(ru 0.000000000s ub 205.841339111s int 16.000000000s)", 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.149707+0000" 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.149722+0000" 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.150026+0000" 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.150209+0000" 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.151202+0000" 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-22T01:58:30.233 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.151346+0000" 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.151398+0000" 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.234 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "peering_event", 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 113, 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "PeeringEvent": { 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "1", 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.0", 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "sent": 51, 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "requested": 51, 2024-04-22T01:58:30.235 INFO:teuthology.orchestra.run.smithi107.stdout: "evt": "epoch_sent: 51 epoch_requested: 51 MNotifyRec 4.0 from 1 notify: (query:51 sent:51 4.0( v 45'48 (0'0,45'48] local-lis/les=48/49 n=0 ec=43/43 lis/c=48/43 les/c/f=49/44/0 sis=50 pruub=13.060609818s) ([43,49] all_participants={0, 1, 3} intervals=[([48,49] acting={1, 3})])) features: 0x3f03cffffffdffff +create_info", 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.154077+0000" 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitActive", 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.154090+0000" 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::AwaitMap", 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.154335+0000" 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.236 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)0>::OSDMapBlocker", 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::ConnectionPipeline::GetPGMapping", 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.154519+0000" 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PerShardPipeline::CreateOrWaitPG", 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.156256+0000" 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGMap::PGCreationState", 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.156493+0000" 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.156572+0000" 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "peering_event", 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 33554490, 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "detail": { 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "PeeringEvent": { 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "from": "0", 2024-04-22T01:58:30.239 INFO:teuthology.orchestra.run.smithi107.stdout: "pgid": "4.c", 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "sent": 51, 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "requested": 51, 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "evt": "epoch_sent: 51 epoch_requested: 51 ActivateCommitted from 51 processed at 51", 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::StartEvent", 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::AwaitMap", 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.157943+0000" 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.240 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::OSDMapGate<(crimson::osd::OSDMapGateType)1>::OSDMapBlocker", 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::PGPeeringPipeline::Process", 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-04-22T01:58:30.158072+0000" 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "crimson::osd::TrackableOperationT::CompletionEvent", 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "0.000000" 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.241 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-04-22T01:58:30.242 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-04-22T01:58:30.242 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:58:30.284 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:30.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:31.139 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 1321 1305 0.188779 0.09375 0.0733431 0.66162 2024-04-22T01:58:32.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 1499 1483 0.210628 1.39062 0.108022 0.592815 2024-04-22T01:58:32.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:32.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:32.447+0000 7f4bf099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:32.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:32.454+0000 7f4bf099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:32.548 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:32.583 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:33.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 1657 1641 0.228906 1.23438 0.0571477 0.545293 2024-04-22T01:58:34.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 1833 1817 0.24901 1.375 0.0395251 0.501537 2024-04-22T01:58:34.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:34.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:34.731+0000 7f384282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:34.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:34.737+0000 7f384282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:34.782 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:34.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:35.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 1980 1964 0.264515 1.14844 0.121516 0.472072 2024-04-22T01:58:35.244 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 1, 0] 2024-04-22T01:58:35.244 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:58:35.244 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T01:58:35.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T01:58:35.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.371+0000 7fd29971d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.385+0000 7fd29971d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:58:35.456 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:58:35.457 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:58:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:35.461 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:35.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T01:58:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.619+0000 7f0f0366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.631+0000 7f0f0366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.677 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:58:35.678 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:58:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:35.682 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:35.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T01:58:35.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.810+0000 7f76f859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.826+0000 7f76f859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.831 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:35.836 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:35.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T01:58:35.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.971+0000 7f4811f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:35.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:35.984+0000 7f4811f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T01:58:36.005 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T01:58:36.006 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T01:58:36.006 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T01:58:36.006 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T01:58:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:36.009 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T01:58:36.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 2136 2120 0.280686 1.21875 0.0868799 0.444913 2024-04-22T01:58:36.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:36.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:36.942+0000 7f470b0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:36.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:36.947+0000 7f470b0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:37.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:37.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:58:37.142014+0000 min lat: 0.0138269 max lat: 46.2866 avg lat: 0.422337 2024-04-22T01:58:37.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:58:37.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 2286 2270 0.295537 1.17188 0.0223825 0.422337 2024-04-22T01:58:38.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 2433 2417 0.309517 1.14844 0.0397799 0.40349 2024-04-22T01:58:39.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:39.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 2580 2564 0.323046 1.14844 0.159838 0.386421 2024-04-22T01:58:39.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:39.167+0000 7fd15395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:39.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:39.174+0000 7fd15395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:39.307 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:39.344 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:40.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 2738 2722 0.33751 1.23438 0.0931744 0.370067 2024-04-22T01:58:41.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 1, 0] 2024-04-22T01:58:41.011 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:58:41.011 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-04-22T01:58:41.011 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T01:58:41.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T01:58:41.014 INFO:tasks.ceph.osd.2:Started 2024-04-22T01:58:41.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T01:58:41.114 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:58:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:41.118 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T01:58:41.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 2902 2886 0.352253 1.28125 0.145089 0.354355 2024-04-22T01:58:41.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:41.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:41.468+0000 7f2b6662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:41.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:41.475+0000 7f2b6662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:41.622 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:41.663 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:42.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 3065 3049 0.366423 1.27344 0.0637011 0.340809 2024-04-22T01:58:43.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 3225 3209 0.379809 1.25 0.0637191 0.3288 2024-04-22T01:58:43.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:43.808+0000 7fbd8955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:43.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:43.816+0000 7fbd8955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:43.912 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:43.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:44.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 3362 3346 0.390113 1.07031 0.147971 0.320116 2024-04-22T01:58:45.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 3518 3502 0.402297 1.21875 0.201526 0.310417 2024-04-22T01:58:45.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:46.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:46.101+0000 7f5df039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:46.108+0000 7f5df039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:46.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T01:58:46.140 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 3660 3644 0.412543 1.10938 0.0652571 0.302719 2024-04-22T01:58:46.189 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-04-22T01:58:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:58:46.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:58:46.238 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:58:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:58:46.242 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-04-22T01:58:47.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 3781 3765 0.420153 0.945312 0.248587 0.29691 2024-04-22T01:58:48.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 3797 3781 0.415995 0.125 0.0659264 0.296063 2024-04-22T01:58:48.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:58:48.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:48.340+0000 7f1ec499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:48.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:48.345+0000 7f1ec499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:49.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 3797 3781 0.410218 0 - 0.296063 2024-04-22T01:58:50.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 3798 3782 0.404706 0.00390625 2.45441 0.296633 2024-04-22T01:58:50.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:58:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:50.547+0000 7f95be59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:50.553+0000 7f95be59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:51.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 3798 3782 0.399237 0 - 0.296633 2024-04-22T01:58:51.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-22T01:58:51.345 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-22T01:58:51.345 INFO:teuthology.orchestra.run.smithi150.stdout: "ops_in_flight": [] 2024-04-22T01:58:51.345 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-22T01:58:52.141 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 3798 3782 0.393913 0 - 0.296633 2024-04-22T01:58:53.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 3798 3782 0.38873 0 - 0.296633 2024-04-22T01:58:54.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 3798 3782 0.383682 0 - 0.296633 2024-04-22T01:58:54.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-22T01:58:55.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 3798 3782 0.378762 0 - 0.296633 2024-04-22T01:58:56.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 3798 3782 0.373968 0 - 0.296633 2024-04-22T01:58:56.356 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 1, 0, 2] 2024-04-22T01:58:56.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T01:58:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:56.474+0000 7eff29b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:56.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:56.484+0000 7eff29b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T01:58:57.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T01:58:57.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:58:57.144061+0000 min lat: 0.0138269 max lat: 46.2866 avg lat: 0.296633 2024-04-22T01:58:57.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:58:57.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 3798 3782 0.369293 0 - 0.296633 2024-04-22T01:58:57.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:57.182+0000 7f726015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:57.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:57.192+0000 7f726015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:58.055 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T01:58:58.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T01:58:58.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 3798 3782 0.364734 0 - 0.296633 2024-04-22T01:58:58.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:58.199+0000 7f531959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:58.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:58.209+0000 7f531959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:59.065 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T01:58:59.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T01:58:59.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 3798 3782 0.360286 0 - 0.296633 2024-04-22T01:58:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:59.199+0000 7f15b526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:58:59.209+0000 7f15b526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:59.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2024-04-22T01:59:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T01:59:00.080 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T01:59:00.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:00.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 3798 3782 0.355945 0 - 0.296633 2024-04-22T01:59:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.190+0000 7f7ed64ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.197+0000 7f7ed64ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:00.381 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:00.381 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:00.394 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-22T01:59:00.229461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179152,"kb_used_data":41520,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308208,"statfs":{"total":386547056640,"available":386363604992,"internally_reserved":0,"allocated":42516480,"data_stored":88291854,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001150"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":99,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.081086+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:58:57.081086+0000","last_peered":"2024-04-22T01:58:57.081086+0000","last_clean":"2024-04-22T01:58:57.081086+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:58:57.081086+0000","last_undegraded":"2024-04-22T01:58:57.081086+0000","last_fullsized":"2024-04-22T01:58:57.081086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":343,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.387825+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:58:57.387825+0000","last_peered":"2024-04-22T01:58:57.387825+0000","last_clean":"2024-04-22T01:58:57.387825+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:58:57.387825+0000","last_undegraded":"2024-04-22T01:58:57.387825+0000","last_fullsized":"2024-04-22T01:58:57.387825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":275,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.113224+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:58.113224+0000","last_undegraded":"2024-04-22T01:58:58.113224+0000","last_fullsized":"2024-04-22T01:58:58.113224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":283,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:58.235647+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:58:58.235647+0000","last_peered":"2024-04-22T01:58:58.235647+0000","last_clean":"2024-04-22T01:58:58.235647+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:58:58.235647+0000","last_undegraded":"2024-04-22T01:58:58.235647+0000","last_fullsized":"2024-04-22T01:58:58.235647+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":348,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.118951+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:58:59.118951+0000","last_peered":"2024-04-22T01:58:59.118951+0000","last_clean":"2024-04-22T01:58:59.118951+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:58:59.118951+0000","last_undegraded":"2024-04-22T01:58:59.118951+0000","last_fullsized":"2024-04-22T01:58:59.118951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.122569+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:58:59.122569+0000","last_peered":"2024-04-22T01:58:59.122569+0000","last_clean":"2024-04-22T01:58:59.122569+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:58:59.122569+0000","last_undegraded":"2024-04-22T01:58:59.122569+0000","last_fullsized":"2024-04-22T01:58:59.122569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":148,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:58:59.120174+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:58:59.120174+0000","last_peered":"2024-04-22T01:58:59.120174+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:58:59.120174+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.123546+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:58:59.123546+0000","last_peered":"2024-04-22T01:58:59.123546+0000","last_clean":"2024-04-22T01:58:59.123546+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:58:59.123546+0000","last_undegraded":"2024-04-22T01:58:59.123546+0000","last_fullsized":"2024-04-22T01:58:59.123546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.067583+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:58:57.067583+0000","last_peered":"2024-04-22T01:58:57.067583+0000","last_clean":"2024-04-22T01:58:57.067583+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:58:57.067583+0000","last_undegraded":"2024-04-22T01:58:57.067583+0000","last_fullsized":"2024-04-22T01:58:57.067583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":339,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.102347+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:58.102347+0000","last_undegraded":"2024-04-22T01:58:58.102347+0000","last_fullsized":"2024-04-22T01:58:58.102347+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.394390+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:58:57.394390+0000","last_peered":"2024-04-22T01:58:57.394390+0000","last_clean":"2024-04-22T01:58:57.394390+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:58:57.394390+0000","last_undegraded":"2024-04-22T01:58:57.394390+0000","last_fullsized":"2024-04-22T01:58:57.394390+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":274,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.116214+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:58:59.116214+0000","last_peered":"2024-04-22T01:58:59.116214+0000","last_clean":"2024-04-22T01:58:59.116214+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:58:59.116214+0000","last_undegraded":"2024-04-22T01:58:59.116214+0000","last_fullsized":"2024-04-22T01:58:59.116214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.078248+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:58:57.078248+0000","last_peered":"2024-04-22T01:58:57.078248+0000","last_clean":"2024-04-22T01:58:57.078248+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:58:57.078248+0000","last_undegraded":"2024-04-22T01:58:57.078248+0000","last_fullsized":"2024-04-22T01:58:57.078248+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39700,"kb_used_data":12876,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332140,"statfs":{"total":96636764160,"available":96596111360,"internally_reserved":0,"allocated":13185024,"data_stored":26066768,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299402,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35012,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336828,"statfs":{"total":96636764160,"available":96600911872,"internally_reserved":0,"allocated":3796992,"data_stored":12160171,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364830,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51756,"kb_used_data":11940,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":12226560,"data_stored":24168289,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:00.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.499+0000 7f09cab1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.506+0000 7f09cab1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:00.692 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:00.693 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:00.705 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-22T01:59:00.229461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179152,"kb_used_data":41520,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308208,"statfs":{"total":386547056640,"available":386363604992,"internally_reserved":0,"allocated":42516480,"data_stored":88291854,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001150"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":99,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.081086+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:58:57.081086+0000","last_peered":"2024-04-22T01:58:57.081086+0000","last_clean":"2024-04-22T01:58:57.081086+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:58:57.081086+0000","last_undegraded":"2024-04-22T01:58:57.081086+0000","last_fullsized":"2024-04-22T01:58:57.081086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":343,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.387825+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:58:57.387825+0000","last_peered":"2024-04-22T01:58:57.387825+0000","last_clean":"2024-04-22T01:58:57.387825+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:58:57.387825+0000","last_undegraded":"2024-04-22T01:58:57.387825+0000","last_fullsized":"2024-04-22T01:58:57.387825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":275,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.113224+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:58.113224+0000","last_undegraded":"2024-04-22T01:58:58.113224+0000","last_fullsized":"2024-04-22T01:58:58.113224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":283,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:58.235647+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:58:58.235647+0000","last_peered":"2024-04-22T01:58:58.235647+0000","last_clean":"2024-04-22T01:58:58.235647+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:58:58.235647+0000","last_undegraded":"2024-04-22T01:58:58.235647+0000","last_fullsized":"2024-04-22T01:58:58.235647+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":348,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.118951+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:58:59.118951+0000","last_peered":"2024-04-22T01:58:59.118951+0000","last_clean":"2024-04-22T01:58:59.118951+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:58:59.118951+0000","last_undegraded":"2024-04-22T01:58:59.118951+0000","last_fullsized":"2024-04-22T01:58:59.118951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.122569+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:58:59.122569+0000","last_peered":"2024-04-22T01:58:59.122569+0000","last_clean":"2024-04-22T01:58:59.122569+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:58:59.122569+0000","last_undegraded":"2024-04-22T01:58:59.122569+0000","last_fullsized":"2024-04-22T01:58:59.122569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":148,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:58:59.120174+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:58:59.120174+0000","last_peered":"2024-04-22T01:58:59.120174+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:58:59.120174+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.123546+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:58:59.123546+0000","last_peered":"2024-04-22T01:58:59.123546+0000","last_clean":"2024-04-22T01:58:59.123546+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:58:59.123546+0000","last_undegraded":"2024-04-22T01:58:59.123546+0000","last_fullsized":"2024-04-22T01:58:59.123546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.067583+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:58:57.067583+0000","last_peered":"2024-04-22T01:58:57.067583+0000","last_clean":"2024-04-22T01:58:57.067583+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:58:57.067583+0000","last_undegraded":"2024-04-22T01:58:57.067583+0000","last_fullsized":"2024-04-22T01:58:57.067583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":339,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.102347+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:58.102347+0000","last_undegraded":"2024-04-22T01:58:58.102347+0000","last_fullsized":"2024-04-22T01:58:58.102347+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.394390+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:58:57.394390+0000","last_peered":"2024-04-22T01:58:57.394390+0000","last_clean":"2024-04-22T01:58:57.394390+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:58:57.394390+0000","last_undegraded":"2024-04-22T01:58:57.394390+0000","last_fullsized":"2024-04-22T01:58:57.394390+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":274,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.116214+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:58:59.116214+0000","last_peered":"2024-04-22T01:58:59.116214+0000","last_clean":"2024-04-22T01:58:59.116214+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:58:59.116214+0000","last_undegraded":"2024-04-22T01:58:59.116214+0000","last_fullsized":"2024-04-22T01:58:59.116214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.078248+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:58:57.078248+0000","last_peered":"2024-04-22T01:58:57.078248+0000","last_clean":"2024-04-22T01:58:57.078248+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:58:57.078248+0000","last_undegraded":"2024-04-22T01:58:57.078248+0000","last_fullsized":"2024-04-22T01:58:57.078248+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39700,"kb_used_data":12876,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332140,"statfs":{"total":96636764160,"available":96596111360,"internally_reserved":0,"allocated":13185024,"data_stored":26066768,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299402,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35012,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336828,"statfs":{"total":96636764160,"available":96600911872,"internally_reserved":0,"allocated":3796992,"data_stored":12160171,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364830,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51756,"kb_used_data":11940,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":12226560,"data_stored":24168289,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:00.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.811+0000 7f692555d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:00.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:00.818+0000 7f692555d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:01.068 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 116/1173 objects degraded (9.889%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":225,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1713751127,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"activating+undersized+remapped","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":391,"data_bytes":26075680,"bytes_used":183427072,"bytes_avail":386363629568,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"degraded_objects":116,"degraded_total":1173,"degraded_ratio":0.098891730605285597,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":95360,"recovering_keys_per_sec":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:01.082 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-22T01:59:01.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:01.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 3798 3782 0.351708 0 - 0.296633 2024-04-22T01:59:01.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:01.185+0000 7f600c3d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:01.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:01.193+0000 7f600c3d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:01.377 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:01.378 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:01.390 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-04-22T01:59:00.229461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179152,"kb_used_data":41520,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308208,"statfs":{"total":386547056640,"available":386363604992,"internally_reserved":0,"allocated":42516480,"data_stored":88291854,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001150"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":99,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.081086+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:58:57.081086+0000","last_peered":"2024-04-22T01:58:57.081086+0000","last_clean":"2024-04-22T01:58:57.081086+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:58:57.081086+0000","last_undegraded":"2024-04-22T01:58:57.081086+0000","last_fullsized":"2024-04-22T01:58:57.081086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":343,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.387825+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:58:57.387825+0000","last_peered":"2024-04-22T01:58:57.387825+0000","last_clean":"2024-04-22T01:58:57.387825+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:58:57.387825+0000","last_undegraded":"2024-04-22T01:58:57.387825+0000","last_fullsized":"2024-04-22T01:58:57.387825+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":275,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.113224+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:58.113224+0000","last_undegraded":"2024-04-22T01:58:58.113224+0000","last_fullsized":"2024-04-22T01:58:58.113224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":283,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:58.235647+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:58:58.235647+0000","last_peered":"2024-04-22T01:58:58.235647+0000","last_clean":"2024-04-22T01:58:58.235647+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:58:58.235647+0000","last_undegraded":"2024-04-22T01:58:58.235647+0000","last_fullsized":"2024-04-22T01:58:58.235647+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":348,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.118951+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:58:59.118951+0000","last_peered":"2024-04-22T01:58:59.118951+0000","last_clean":"2024-04-22T01:58:59.118951+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:58:59.118951+0000","last_undegraded":"2024-04-22T01:58:59.118951+0000","last_fullsized":"2024-04-22T01:58:59.118951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.122569+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:58:59.122569+0000","last_peered":"2024-04-22T01:58:59.122569+0000","last_clean":"2024-04-22T01:58:59.122569+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:58:59.122569+0000","last_undegraded":"2024-04-22T01:58:59.122569+0000","last_fullsized":"2024-04-22T01:58:59.122569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":148,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:58:59.120174+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:58:59.120174+0000","last_peered":"2024-04-22T01:58:59.120174+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:58:59.120174+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.123546+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:58:59.123546+0000","last_peered":"2024-04-22T01:58:59.123546+0000","last_clean":"2024-04-22T01:58:59.123546+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:58:59.123546+0000","last_undegraded":"2024-04-22T01:58:59.123546+0000","last_fullsized":"2024-04-22T01:58:59.123546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.067583+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:58:57.067583+0000","last_peered":"2024-04-22T01:58:57.067583+0000","last_clean":"2024-04-22T01:58:57.067583+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:58:57.067583+0000","last_undegraded":"2024-04-22T01:58:57.067583+0000","last_fullsized":"2024-04-22T01:58:57.067583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":339,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.102347+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:58.102347+0000","last_undegraded":"2024-04-22T01:58:58.102347+0000","last_fullsized":"2024-04-22T01:58:58.102347+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":144,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.394390+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:58:57.394390+0000","last_peered":"2024-04-22T01:58:57.394390+0000","last_clean":"2024-04-22T01:58:57.394390+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:58:57.394390+0000","last_undegraded":"2024-04-22T01:58:57.394390+0000","last_fullsized":"2024-04-22T01:58:57.394390+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":274,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.116214+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:58:59.116214+0000","last_peered":"2024-04-22T01:58:59.116214+0000","last_clean":"2024-04-22T01:58:59.116214+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:58:59.116214+0000","last_undegraded":"2024-04-22T01:58:59.116214+0000","last_fullsized":"2024-04-22T01:58:59.116214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":85,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-04-22T01:58:57.078248+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:58:57.078248+0000","last_peered":"2024-04-22T01:58:57.078248+0000","last_clean":"2024-04-22T01:58:57.078248+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:58:57.078248+0000","last_undegraded":"2024-04-22T01:58:57.078248+0000","last_fullsized":"2024-04-22T01:58:57.078248+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39700,"kb_used_data":12876,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332140,"statfs":{"total":96636764160,"available":96596111360,"internally_reserved":0,"allocated":13185024,"data_stored":26066768,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299402,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35012,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336828,"statfs":{"total":96636764160,"available":96600911872,"internally_reserved":0,"allocated":3796992,"data_stored":12160171,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364830,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51756,"kb_used_data":11940,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":12226560,"data_stored":24168289,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:02.142 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 3798 3782 0.34757 0 - 0.296633 2024-04-22T01:59:03.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 3798 3782 0.343529 0 - 0.296633 2024-04-22T01:59:04.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 3798 3782 0.33958 0 - 0.296633 2024-04-22T01:59:04.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:04.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:04.510+0000 7f5da246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:04.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:04.518+0000 7f5da246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:04.705 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:04.718 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-04-22T01:59:04.230355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179208,"kb_used_data":41576,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308152,"statfs":{"total":386547056640,"available":386363547648,"internally_reserved":0,"allocated":42573824,"data_stored":88315787,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.172641"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":275,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.113224+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:58.113224+0000","last_undegraded":"2024-04-22T01:58:58.113224+0000","last_fullsized":"2024-04-22T01:58:58.113224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":348,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.118951+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:58:59.118951+0000","last_peered":"2024-04-22T01:58:59.118951+0000","last_clean":"2024-04-22T01:58:59.118951+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:58:59.118951+0000","last_undegraded":"2024-04-22T01:58:59.118951+0000","last_fullsized":"2024-04-22T01:58:59.118951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.122569+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:58:59.122569+0000","last_peered":"2024-04-22T01:58:59.122569+0000","last_clean":"2024-04-22T01:58:59.122569+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:58:59.122569+0000","last_undegraded":"2024-04-22T01:58:59.122569+0000","last_fullsized":"2024-04-22T01:58:59.122569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":148,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:58:59.120174+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:58:59.120174+0000","last_peered":"2024-04-22T01:58:59.120174+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:58:59.120174+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.123546+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:58:59.123546+0000","last_peered":"2024-04-22T01:58:59.123546+0000","last_clean":"2024-04-22T01:58:59.123546+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:58:59.123546+0000","last_undegraded":"2024-04-22T01:58:59.123546+0000","last_fullsized":"2024-04-22T01:58:59.123546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":339,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.102347+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:58.102347+0000","last_undegraded":"2024-04-22T01:58:58.102347+0000","last_fullsized":"2024-04-22T01:58:58.102347+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":274,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.116214+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:58:59.116214+0000","last_peered":"2024-04-22T01:58:59.116214+0000","last_clean":"2024-04-22T01:58:59.116214+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:58:59.116214+0000","last_undegraded":"2024-04-22T01:58:59.116214+0000","last_fullsized":"2024-04-22T01:58:59.116214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299407,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364830,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51756,"kb_used_data":11940,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":12226560,"data_stored":24168289,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:04.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:04.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:04.822+0000 7ff1afe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:04.829+0000 7ff1afe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:05.086 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:05.086 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 116/1173 objects degraded (9.889%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":229,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1713751127,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"activating+undersized+remapped","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":391,"data_bytes":26075680,"bytes_used":183508992,"bytes_avail":386363547648,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"degraded_objects":116,"degraded_total":1173,"degraded_ratio":0.098891730605285597},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:05.099 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-22T01:59:05.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:05.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 3798 3782 0.335721 0 - 0.296633 2024-04-22T01:59:05.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-22T01:59:05.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:05.202+0000 7f794c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:05.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:05.210+0000 7f794c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:05.397 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:05.398 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:05.410 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-04-22T01:59:04.230355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179208,"kb_used_data":41576,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308152,"statfs":{"total":386547056640,"available":386363547648,"internally_reserved":0,"allocated":42573824,"data_stored":88315787,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.172641"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":275,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.113224+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:58.113224+0000","last_undegraded":"2024-04-22T01:58:58.113224+0000","last_fullsized":"2024-04-22T01:58:58.113224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":348,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.118951+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:58:59.118951+0000","last_peered":"2024-04-22T01:58:59.118951+0000","last_clean":"2024-04-22T01:58:59.118951+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:58:59.118951+0000","last_undegraded":"2024-04-22T01:58:59.118951+0000","last_fullsized":"2024-04-22T01:58:59.118951+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.122569+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:58:59.122569+0000","last_peered":"2024-04-22T01:58:59.122569+0000","last_clean":"2024-04-22T01:58:59.122569+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:58:59.122569+0000","last_undegraded":"2024-04-22T01:58:59.122569+0000","last_fullsized":"2024-04-22T01:58:59.122569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":148,"reported_epoch":60,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:58:59.120174+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:58:59.120174+0000","last_peered":"2024-04-22T01:58:59.120174+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:58:59.120174+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.123546+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:58:59.123546+0000","last_peered":"2024-04-22T01:58:59.123546+0000","last_clean":"2024-04-22T01:58:59.123546+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:58:59.123546+0000","last_undegraded":"2024-04-22T01:58:59.123546+0000","last_fullsized":"2024-04-22T01:58:59.123546+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":339,"reported_epoch":60,"state":"peering","last_fresh":"2024-04-22T01:58:58.102347+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:58.102347+0000","last_undegraded":"2024-04-22T01:58:58.102347+0000","last_fullsized":"2024-04-22T01:58:58.102347+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":274,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T01:58:59.116214+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:58:59.116214+0000","last_peered":"2024-04-22T01:58:59.116214+0000","last_clean":"2024-04-22T01:58:59.116214+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:58:59.116214+0000","last_undegraded":"2024-04-22T01:58:59.116214+0000","last_fullsized":"2024-04-22T01:58:59.116214+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299407,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364830,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51756,"kb_used_data":11940,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":12226560,"data_stored":24168289,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:06.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 3798 3782 0.331949 0 - 0.296633 2024-04-22T01:59:07.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 16 3798 3782 0.328261 0 - 0.296633 2024-04-22T01:59:08.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 91 16 3798 3782 0.324654 0 - 0.296633 2024-04-22T01:59:08.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:08.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:08.529+0000 7f3469b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:08.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:08.537+0000 7f3469b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:08.724 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:08.737 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T01:59:08.231539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.164290"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299412,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364835,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:08.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:08.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:08.842+0000 7f3e314ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:08.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:08.848+0000 7f3e314ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:09.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:09.103 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 116/1173 objects degraded (9.889%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":233,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1713751127,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"activating+undersized+remapped","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":391,"data_bytes":26075680,"bytes_used":183517184,"bytes_avail":386363539456,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"degraded_objects":116,"degraded_total":1173,"degraded_ratio":0.098891730605285597},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:09.116 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-22T01:59:09.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:09.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 92 16 3798 3782 0.321125 0 - 0.296633 2024-04-22T01:59:09.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:09.219+0000 7fdead6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:09.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:09.227+0000 7fdead6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:09.413 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:09.426 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-04-22T01:59:08.231539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.164290"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299412,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364835,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:10.143 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 93 16 3798 3782 0.317672 0 - 0.296633 2024-04-22T01:59:10.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2024-04-22T01:59:11.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 94 16 3798 3782 0.314292 0 - 0.296633 2024-04-22T01:59:12.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 95 16 3798 3782 0.310984 0 - 0.296633 2024-04-22T01:59:12.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:12.544+0000 7f74726ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:12.552+0000 7f74726ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:12.750 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-04-22T01:59:12.232663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299412,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:12.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:12.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:12.855+0000 7ff5f1d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:12.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:12.861+0000 7ff5f1d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:13.116 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:13.116 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 116/1173 objects degraded (9.889%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1713751127,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"activating+undersized+remapped","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":391,"data_bytes":26075680,"bytes_used":183517184,"bytes_avail":386363539456,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"degraded_objects":116,"degraded_total":1173,"degraded_ratio":0.098891730605285597},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:13.129 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-22T01:59:13.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:13.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 96 16 3798 3782 0.307745 0 - 0.296633 2024-04-22T01:59:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:13.232+0000 7f89968ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:13.240+0000 7f89968ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:13.424 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:13.425 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:13.438 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-04-22T01:59:12.232663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003201"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299412,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364840,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:14.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 97 16 3798 3782 0.304572 0 - 0.296633 2024-04-22T01:59:15.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 98 16 3798 3782 0.301464 0 - 0.296633 2024-04-22T01:59:16.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 99 16 3798 3782 0.298419 0 - 0.296633 2024-04-22T01:59:16.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-04-22T01:59:16.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:16.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:16.556+0000 7f508166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:16.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:16.563+0000 7f508166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:16.748 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:16.761 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-04-22T01:59:16.233769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003414"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:58:49.088098+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:58:49.049255+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:58:49.097542+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:58:49.077480+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:58:49.095945+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:58:49.072410+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:58:49.078417+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:58:49.080367+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:58:49.108331+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:58:49.059984+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:58:49.047254+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:58:49.081829+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299417,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364845,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:16.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:16.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:16.865+0000 7f0caf59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:16.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:16.872+0000 7f0caf59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:17.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:59:17.146345+0000 min lat: 0.0138269 max lat: 46.2866 avg lat: 0.296633 2024-04-22T01:59:17.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:59:17.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 100 16 3798 3782 0.295435 0 - 0.296633 2024-04-22T01:59:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:17.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:17.210 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:17.210 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 116/1173 objects degraded (9.889%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":241,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":62,"num_osds":4,"num_up_osds":3,"osd_up_since":1713751157,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"activating+undersized+remapped","count":5},{"state_name":"peering","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":391,"data_bytes":26075680,"bytes_used":183517184,"bytes_avail":386363539456,"bytes_total":386547056640,"inactive_pgs_ratio":0.2800000011920929,"degraded_objects":116,"degraded_total":1173,"degraded_ratio":0.098891730605285597},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:17.230 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-22T01:59:17.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:17.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:17.379+0000 7ff631b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:17.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:17.387+0000 7ff631b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:17.575 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:17.576 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:17.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-22T01:59:17.050245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26075680,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3248,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3880,"ondisk_log_size":3880,"up":75,"acting":69,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179216,"kb_used_data":41584,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308144,"statfs":{"total":386547056640,"available":386363539456,"internally_reserved":0,"allocated":42582016,"data_stored":88319206,"data_compressed":4495165,"data_compressed_allocated":40554496,"data_compressed_original":87523328,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.819364"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.126664+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:00.126664+0000","last_peered":"2024-04-22T01:59:00.126664+0000","last_clean":"2024-04-22T01:59:00.126664+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:00.126664+0000","last_undegraded":"2024-04-22T01:59:00.126664+0000","last_fullsized":"2024-04-22T01:59:00.126664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:59:17.050290+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":264,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.049255+0000","last_change":"2024-04-22T01:58:49.058218+0000","last_active":"2024-04-22T01:58:47.071498+0000","last_peered":"2024-04-22T01:58:46.388104+0000","last_clean":"2024-04-22T01:57:43.259206+0000","last_became_active":"2024-04-22T01:58:30.252313+0000","last_became_peered":"2024-04-22T01:58:30.252313+0000","last_unstale":"2024-04-22T01:59:17.050294+0000","last_undegraded":"2024-04-22T01:58:49.049255+0000","last_fullsized":"2024-04-22T01:58:49.049255+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":232,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":352,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.086157+0000","last_change":"2024-04-22T01:58:30.201990+0000","last_active":"2024-04-22T01:59:00.086157+0000","last_peered":"2024-04-22T01:59:00.086157+0000","last_clean":"2024-04-22T01:59:00.086157+0000","last_became_active":"2024-04-22T01:58:30.197446+0000","last_became_peered":"2024-04-22T01:58:30.197446+0000","last_unstale":"2024-04-22T01:59:00.086157+0000","last_undegraded":"2024-04-22T01:59:00.086157+0000","last_fullsized":"2024-04-22T01:59:00.086157+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":276,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.120545+0000","last_change":"2024-04-22T01:58:51.067167+0000","last_active":"2024-04-22T01:58:51.067167+0000","last_peered":"2024-04-22T01:58:50.173783+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:58:49.322405+0000","last_became_peered":"2024-04-22T01:58:49.322405+0000","last_unstale":"2024-04-22T01:58:59.120545+0000","last_undegraded":"2024-04-22T01:58:59.120545+0000","last_fullsized":"2024-04-22T01:58:59.120545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":228,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.097542+0000","last_change":"2024-04-22T01:58:49.105545+0000","last_active":"2024-04-22T01:58:47.082895+0000","last_peered":"2024-04-22T01:58:46.560099+0000","last_clean":"2024-04-22T01:57:44.455381+0000","last_became_active":"2024-04-22T01:58:13.128560+0000","last_became_peered":"2024-04-22T01:58:13.128560+0000","last_unstale":"2024-04-22T01:59:17.050298+0000","last_undegraded":"2024-04-22T01:58:49.097542+0000","last_fullsized":"2024-04-22T01:58:49.097542+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":243,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.077480+0000","last_change":"2024-04-22T01:58:30.284590+0000","last_active":"2024-04-22T01:58:49.077480+0000","last_peered":"2024-04-22T01:58:49.077480+0000","last_clean":"2024-04-22T01:58:49.077480+0000","last_became_active":"2024-04-22T01:58:30.275798+0000","last_became_peered":"2024-04-22T01:58:30.275798+0000","last_unstale":"2024-04-22T01:59:17.050300+0000","last_undegraded":"2024-04-22T01:58:49.077480+0000","last_fullsized":"2024-04-22T01:58:49.077480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":347,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.095945+0000","last_change":"2024-04-22T01:58:30.302207+0000","last_active":"2024-04-22T01:58:49.095945+0000","last_peered":"2024-04-22T01:58:49.095945+0000","last_clean":"2024-04-22T01:58:49.095945+0000","last_became_active":"2024-04-22T01:58:30.295440+0000","last_became_peered":"2024-04-22T01:58:30.295440+0000","last_unstale":"2024-04-22T01:59:17.050303+0000","last_undegraded":"2024-04-22T01:58:49.095945+0000","last_fullsized":"2024-04-22T01:58:49.095945+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":243,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.072410+0000","last_change":"2024-04-22T01:58:30.241752+0000","last_active":"2024-04-22T01:58:49.072410+0000","last_peered":"2024-04-22T01:58:49.072410+0000","last_clean":"2024-04-22T01:58:49.072410+0000","last_became_active":"2024-04-22T01:58:30.235095+0000","last_became_peered":"2024-04-22T01:58:30.235095+0000","last_unstale":"2024-04-22T01:59:17.050305+0000","last_undegraded":"2024-04-22T01:58:49.072410+0000","last_fullsized":"2024-04-22T01:58:49.072410+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":289,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.085156+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:00.085156+0000","last_peered":"2024-04-22T01:59:00.085156+0000","last_clean":"2024-04-22T01:59:00.085156+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:00.085156+0000","last_undegraded":"2024-04-22T01:59:00.085156+0000","last_fullsized":"2024-04-22T01:59:00.085156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.078417+0000","last_change":"2024-04-22T01:58:48.192979+0000","last_active":"2024-04-22T01:58:49.078417+0000","last_peered":"2024-04-22T01:58:49.078417+0000","last_clean":"2024-04-22T01:58:49.078417+0000","last_became_active":"2024-04-22T01:58:48.179288+0000","last_became_peered":"2024-04-22T01:58:48.179288+0000","last_unstale":"2024-04-22T01:59:17.050308+0000","last_undegraded":"2024-04-22T01:58:49.078417+0000","last_fullsized":"2024-04-22T01:58:49.078417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":351,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.159925+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:00.159925+0000","last_peered":"2024-04-22T01:59:00.159925+0000","last_clean":"2024-04-22T01:59:00.159925+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:00.159925+0000","last_undegraded":"2024-04-22T01:59:00.159925+0000","last_fullsized":"2024-04-22T01:59:00.159925+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.160101+0000","last_change":"2024-04-22T01:58:48.204243+0000","last_active":"2024-04-22T01:59:00.160101+0000","last_peered":"2024-04-22T01:59:00.160101+0000","last_clean":"2024-04-22T01:59:00.160101+0000","last_became_active":"2024-04-22T01:58:48.198724+0000","last_became_peered":"2024-04-22T01:58:48.198724+0000","last_unstale":"2024-04-22T01:59:00.160101+0000","last_undegraded":"2024-04-22T01:59:00.160101+0000","last_fullsized":"2024-04-22T01:59:00.160101+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.080367+0000","last_change":"2024-04-22T01:58:48.160326+0000","last_active":"2024-04-22T01:58:49.080367+0000","last_peered":"2024-04-22T01:58:49.080367+0000","last_clean":"2024-04-22T01:58:49.080367+0000","last_became_active":"2024-04-22T01:58:48.155737+0000","last_became_peered":"2024-04-22T01:58:48.155737+0000","last_unstale":"2024-04-22T01:59:17.050311+0000","last_undegraded":"2024-04-22T01:58:49.080367+0000","last_fullsized":"2024-04-22T01:58:49.080367+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:17.050314+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.158074+0000","last_change":"2024-04-22T01:58:48.201908+0000","last_active":"2024-04-22T01:59:00.158074+0000","last_peered":"2024-04-22T01:59:00.158074+0000","last_clean":"2024-04-22T01:59:00.158074+0000","last_became_active":"2024-04-22T01:58:48.193533+0000","last_became_peered":"2024-04-22T01:58:48.193533+0000","last_unstale":"2024-04-22T01:59:00.158074+0000","last_undegraded":"2024-04-22T01:59:00.158074+0000","last_fullsized":"2024-04-22T01:59:00.158074+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":236,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.059984+0000","last_change":"2024-04-22T01:58:49.068600+0000","last_active":"2024-04-22T01:58:47.073725+0000","last_peered":"2024-04-22T01:58:46.453687+0000","last_clean":"2024-04-22T01:57:43.177732+0000","last_became_active":"2024-04-22T01:58:13.107007+0000","last_became_peered":"2024-04-22T01:58:13.107007+0000","last_unstale":"2024-04-22T01:59:17.050322+0000","last_undegraded":"2024-04-22T01:58:49.059984+0000","last_fullsized":"2024-04-22T01:58:49.059984+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.120330+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:00.120330+0000","last_peered":"2024-04-22T01:59:00.120330+0000","last_clean":"2024-04-22T01:59:00.120330+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:00.120330+0000","last_undegraded":"2024-04-22T01:59:00.120330+0000","last_fullsized":"2024-04-22T01:59:00.120330+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"54'263","reported_seq":340,"reported_epoch":61,"state":"peering","last_fresh":"2024-04-22T01:58:59.107384+0000","last_change":"2024-04-22T01:58:50.058836+0000","last_active":"2024-04-22T01:58:50.058836+0000","last_peered":"2024-04-22T01:58:49.724671+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:58:49.356627+0000","last_became_peered":"2024-04-22T01:58:49.356627+0000","last_unstale":"2024-04-22T01:58:59.107384+0000","last_undegraded":"2024-04-22T01:58:59.107384+0000","last_fullsized":"2024-04-22T01:58:59.107384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2154496,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":2104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":153,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.096919+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:00.096919+0000","last_peered":"2024-04-22T01:59:00.096919+0000","last_clean":"2024-04-22T01:59:00.096919+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:00.096919+0000","last_undegraded":"2024-04-22T01:59:00.096919+0000","last_fullsized":"2024-04-22T01:59:00.096919+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":277,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.162851+0000","last_change":"2024-04-22T01:58:30.338714+0000","last_active":"2024-04-22T01:59:00.162851+0000","last_peered":"2024-04-22T01:59:00.162851+0000","last_clean":"2024-04-22T01:59:00.162851+0000","last_became_active":"2024-04-22T01:58:30.333191+0000","last_became_peered":"2024-04-22T01:58:30.333191+0000","last_unstale":"2024-04-22T01:59:00.162851+0000","last_undegraded":"2024-04-22T01:59:00.162851+0000","last_fullsized":"2024-04-22T01:59:00.162851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-04-22T01:59:00.129961+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:00.129961+0000","last_peered":"2024-04-22T01:59:00.129961+0000","last_clean":"2024-04-22T01:59:00.129961+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:00.129961+0000","last_undegraded":"2024-04-22T01:59:00.129961+0000","last_fullsized":"2024-04-22T01:59:00.129961+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":352,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.047254+0000","last_change":"2024-04-22T01:58:30.427456+0000","last_active":"2024-04-22T01:58:49.047254+0000","last_peered":"2024-04-22T01:58:49.047254+0000","last_clean":"2024-04-22T01:58:49.047254+0000","last_became_active":"2024-04-22T01:58:30.220655+0000","last_became_peered":"2024-04-22T01:58:30.220655+0000","last_unstale":"2024-04-22T01:59:17.050325+0000","last_undegraded":"2024-04-22T01:58:49.047254+0000","last_fullsized":"2024-04-22T01:58:49.047254+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":54,"state":"stale+active+clean","last_fresh":"2024-04-22T01:58:49.081829+0000","last_change":"2024-04-22T01:58:48.172311+0000","last_active":"2024-04-22T01:58:49.081829+0000","last_peered":"2024-04-22T01:58:49.081829+0000","last_clean":"2024-04-22T01:58:49.081829+0000","last_became_active":"2024-04-22T01:58:48.164849+0000","last_became_peered":"2024-04-22T01:58:48.164849+0000","last_unstale":"2024-04-22T01:59:17.050328+0000","last_undegraded":"2024-04-22T01:58:49.081829+0000","last_fullsized":"2024-04-22T01:58:49.081829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":25485312,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":5218304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3783,"ondisk_log_size":3783,"up":48,"acting":42,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39732,"kb_used_data":12908,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":13217792,"data_stored":26080444,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299417,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35036,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336804,"statfs":{"total":96636764160,"available":96600887296,"internally_reserved":0,"allocated":3821568,"data_stored":12170428,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364845,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51764,"kb_used_data":11948,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320076,"statfs":{"total":96636764160,"available":96583757824,"internally_reserved":0,"allocated":12234752,"data_stored":24171708,"data_compressed":1236548,"data_compressed_allocated":11706368,"data_compressed_original":23961600,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:18.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 101 16 3798 3782 0.29251 0 - 0.296633 2024-04-22T01:59:19.144 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 102 6 3798 3792 0.290408 0.00269397 31.3079 0.378347 2024-04-22T01:59:19.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:19.311+0000 7f684666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:19.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:19.317+0000 7f684666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:19.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 102.187 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 3798 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 8192 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 0.290367 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 0.508381 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 1.39062 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 37 2024-04-22T01:59:20.145 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 65.0802 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 178 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.427601 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 3.0508 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 46.2866 2024-04-22T01:59:20.146 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.0138269 2024-04-22T01:59:20.170 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-04-22T01:59:20.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:20.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:20.709+0000 7f0a02a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:20.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:20.718+0000 7f0a02a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:20.944 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:20.944 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:20.962 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-04-22T01:59:20.234689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12780064,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1625,"num_write_kb":13798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6266880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3889,"ondisk_log_size":3889,"up":59,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179300,"kb_used_data":41668,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308060,"statfs":{"total":386547056640,"available":386363453440,"internally_reserved":0,"allocated":42668032,"data_stored":88427469,"data_compressed":4499818,"data_compressed_allocated":40599552,"data_compressed_original":87613440,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13295616,"num_objects":-203,"num_object_clones":0,"num_object_copies":-609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1623,"num_write_kb":-12984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001539"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.104797+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:17.104797+0000","last_peered":"2024-04-22T01:59:17.104797+0000","last_clean":"2024-04-22T01:59:17.104797+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:17.104797+0000","last_undegraded":"2024-04-22T01:59:17.104797+0000","last_fullsized":"2024-04-22T01:59:17.104797+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:59:17.050290+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":47,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.549027+0000","last_change":"2024-04-22T01:59:19.550741+0000","last_active":"2024-04-22T01:59:19.549027+0000","last_peered":"2024-04-22T01:59:19.549027+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.122641+0000","last_became_peered":"2024-04-22T01:59:19.122641+0000","last_unstale":"2024-04-22T01:59:19.549027+0000","last_undegraded":"2024-04-22T01:59:19.103724+0000","last_fullsized":"2024-04-22T01:59:19.101613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":359,"reported_epoch":63,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:18.147279+0000","last_change":"2024-04-22T01:59:18.146416+0000","last_active":"2024-04-22T01:59:18.147279+0000","last_peered":"2024-04-22T01:59:18.147279+0000","last_clean":"2024-04-22T01:59:00.086529+0000","last_became_active":"2024-04-22T01:59:18.146416+0000","last_became_peered":"2024-04-22T01:59:18.146416+0000","last_unstale":"2024-04-22T01:59:18.147279+0000","last_undegraded":"2024-04-22T01:59:18.075324+0000","last_fullsized":"2024-04-22T01:59:18.073778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":286,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.101558+0000","last_change":"2024-04-22T01:59:18.200119+0000","last_active":"2024-04-22T01:59:19.101558+0000","last_peered":"2024-04-22T01:59:19.101558+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:59:18.200119+0000","last_became_peered":"2024-04-22T01:59:18.200119+0000","last_unstale":"2024-04-22T01:59:19.101558+0000","last_undegraded":"2024-04-22T01:59:18.087681+0000","last_fullsized":"2024-04-22T01:59:18.086263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":5,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-22T01:59:18.105700+0000","last_change":"2024-04-22T01:59:18.105700+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:18.105700+0000","last_undegraded":"2024-04-22T01:59:18.105700+0000","last_fullsized":"2024-04-22T01:59:18.105700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":9,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.139804+0000","last_change":"2024-04-22T01:59:18.138497+0000","last_active":"2024-04-22T01:59:18.139804+0000","last_peered":"2024-04-22T01:59:18.139804+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.138497+0000","last_became_peered":"2024-04-22T01:59:18.138497+0000","last_unstale":"2024-04-22T01:59:18.139804+0000","last_undegraded":"2024-04-22T01:59:18.139804+0000","last_fullsized":"2024-04-22T01:59:18.100062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.083651+0000","last_change":"2024-04-22T01:59:18.160401+0000","last_active":"2024-04-22T01:59:19.083651+0000","last_peered":"2024-04-22T01:59:19.083651+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.160401+0000","last_became_peered":"2024-04-22T01:59:18.160401+0000","last_unstale":"2024-04-22T01:59:19.083651+0000","last_undegraded":"2024-04-22T01:59:19.083651+0000","last_fullsized":"2024-04-22T01:59:18.083620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.078848+0000","last_change":"2024-04-22T01:59:18.145130+0000","last_active":"2024-04-22T01:59:19.078848+0000","last_peered":"2024-04-22T01:59:19.078848+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.145130+0000","last_became_peered":"2024-04-22T01:59:18.145130+0000","last_unstale":"2024-04-22T01:59:19.078848+0000","last_undegraded":"2024-04-22T01:59:19.078848+0000","last_fullsized":"2024-04-22T01:59:18.073418+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":295,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:59:18.084023+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:18.084023+0000","last_peered":"2024-04-22T01:59:18.084023+0000","last_clean":"2024-04-22T01:59:18.084023+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:18.084023+0000","last_undegraded":"2024-04-22T01:59:18.084023+0000","last_fullsized":"2024-04-22T01:59:18.084023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.077390+0000","last_change":"2024-04-22T01:59:18.185653+0000","last_active":"2024-04-22T01:59:19.077390+0000","last_peered":"2024-04-22T01:59:19.077390+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:18.185653+0000","last_became_peered":"2024-04-22T01:59:18.185653+0000","last_unstale":"2024-04-22T01:59:19.077390+0000","last_undegraded":"2024-04-22T01:59:19.077390+0000","last_fullsized":"2024-04-22T01:59:18.080636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":360,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.102789+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:19.102789+0000","last_peered":"2024-04-22T01:59:19.102789+0000","last_clean":"2024-04-22T01:59:19.102789+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:19.102789+0000","last_undegraded":"2024-04-22T01:59:19.102789+0000","last_fullsized":"2024-04-22T01:59:19.102789+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.105337+0000","last_change":"2024-04-22T01:59:18.233671+0000","last_active":"2024-04-22T01:59:19.105337+0000","last_peered":"2024-04-22T01:59:19.105337+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:18.233671+0000","last_became_peered":"2024-04-22T01:59:18.233671+0000","last_unstale":"2024-04-22T01:59:19.105337+0000","last_undegraded":"2024-04-22T01:59:19.105337+0000","last_fullsized":"2024-04-22T01:59:18.079371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.182127+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:18.182127+0000","last_peered":"2024-04-22T01:59:18.182127+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:18.182127+0000","last_undegraded":"2024-04-22T01:59:18.182127+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:17.050314+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.097977+0000","last_change":"2024-04-22T01:59:18.230079+0000","last_active":"2024-04-22T01:59:19.097977+0000","last_peered":"2024-04-22T01:59:19.097977+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:18.230079+0000","last_became_peered":"2024-04-22T01:59:18.230079+0000","last_unstale":"2024-04-22T01:59:19.097977+0000","last_undegraded":"2024-04-22T01:59:19.097977+0000","last_fullsized":"2024-04-22T01:59:18.101172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":5,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-22T01:59:18.098361+0000","last_change":"2024-04-22T01:59:18.098361+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:18.098361+0000","last_undegraded":"2024-04-22T01:59:18.098361+0000","last_fullsized":"2024-04-22T01:59:18.098361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":97,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.066184+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:17.066184+0000","last_peered":"2024-04-22T01:59:17.066184+0000","last_clean":"2024-04-22T01:59:17.066184+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:17.066184+0000","last_undegraded":"2024-04-22T01:59:17.066184+0000","last_fullsized":"2024-04-22T01:59:17.066184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"63'272","reported_seq":359,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.088085+0000","last_change":"2024-04-22T01:59:18.176827+0000","last_active":"2024-04-22T01:59:19.088085+0000","last_peered":"2024-04-22T01:59:19.088085+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:59:18.176827+0000","last_became_peered":"2024-04-22T01:59:18.176827+0000","last_unstale":"2024-04-22T01:59:19.088085+0000","last_undegraded":"2024-04-22T01:59:18.071711+0000","last_fullsized":"2024-04-22T01:59:18.070350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":34}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:59:18.105076+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:18.105076+0000","last_peered":"2024-04-22T01:59:18.105076+0000","last_clean":"2024-04-22T01:59:18.105076+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:18.105076+0000","last_undegraded":"2024-04-22T01:59:18.105076+0000","last_fullsized":"2024-04-22T01:59:18.105076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":287,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.098286+0000","last_change":"2024-04-22T01:59:18.183201+0000","last_active":"2024-04-22T01:59:19.098286+0000","last_peered":"2024-04-22T01:59:19.098286+0000","last_clean":"2024-04-22T01:59:00.163168+0000","last_became_active":"2024-04-22T01:59:18.183201+0000","last_became_peered":"2024-04-22T01:59:18.183201+0000","last_unstale":"2024-04-22T01:59:19.098286+0000","last_undegraded":"2024-04-22T01:59:18.081510+0000","last_fullsized":"2024-04-22T01:59:18.080121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.103127+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:17.103127+0000","last_peered":"2024-04-22T01:59:17.103127+0000","last_clean":"2024-04-22T01:59:17.103127+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:17.103127+0000","last_undegraded":"2024-04-22T01:59:17.103127+0000","last_fullsized":"2024-04-22T01:59:17.103127+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":9,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.150784+0000","last_change":"2024-04-22T01:59:18.150038+0000","last_active":"2024-04-22T01:59:18.150784+0000","last_peered":"2024-04-22T01:59:18.150784+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.150038+0000","last_became_peered":"2024-04-22T01:59:18.150038+0000","last_unstale":"2024-04-22T01:59:18.150784+0000","last_undegraded":"2024-04-22T01:59:18.150784+0000","last_fullsized":"2024-04-22T01:59:18.108583+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.107531+0000","last_change":"2024-04-22T01:59:18.209616+0000","last_active":"2024-04-22T01:59:19.107531+0000","last_peered":"2024-04-22T01:59:19.107531+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:18.209616+0000","last_became_peered":"2024-04-22T01:59:18.209616+0000","last_unstale":"2024-04-22T01:59:19.107531+0000","last_undegraded":"2024-04-22T01:59:19.107531+0000","last_fullsized":"2024-04-22T01:59:18.094962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6266880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3792,"ondisk_log_size":3792,"up":37,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39740,"kb_used_data":12916,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332100,"statfs":{"total":96636764160,"available":96596070400,"internally_reserved":0,"allocated":13225984,"data_stored":26084281,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299422,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35044,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336796,"statfs":{"total":96636764160,"available":96600879104,"internally_reserved":0,"allocated":3829760,"data_stored":12174265,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51832,"kb_used_data":12016,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":12304384,"data_stored":24272297,"data_compressed":1241201,"data_compressed_allocated":11751424,"data_compressed_original":24051712,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:20.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-22T01:59:21.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.079+0000 7f12ad6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.087+0000 7f12ad6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:21.362 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"62d4ad10-8dc5-4417-9af4-a9625064a333","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 3 pgs inactive, 2 pgs peering","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 179/564 objects degraded (31.738%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":245,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":64,"num_osds":4,"num_up_osds":3,"osd_up_since":1713751157,"num_in_osds":4,"osd_in_since":1713750918,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":10},{"state_name":"active+clean","count":6},{"state_name":"active+undersized+degraded","count":4},{"state_name":"peering","count":2},{"state_name":"stale+activating+undersized+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":25,"num_pools":3,"num_objects":188,"data_bytes":12780064,"bytes_used":183603200,"bytes_avail":386363453440,"bytes_total":386547056640,"inactive_pgs_ratio":0.15999999642372131,"degraded_objects":179,"degraded_total":564,"degraded_ratio":0.31737588652482268,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":174717,"recovering_keys_per_sec":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-04-22T01:56:22.191956+0000","services":{}},"progress_events":{}} 2024-04-22T01:59:21.377 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-22T01:59:21.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:21.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:21.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.504+0000 7f950315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.513+0000 7f950315d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.625+0000 7f26adead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:21.631+0000 7f26adead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:21.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2024-04-22T01:59:21.751 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:21.767 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-04-22T01:59:20.234689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":12780064,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1625,"num_write_kb":13798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6266880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3889,"ondisk_log_size":3889,"up":59,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":179300,"kb_used_data":41668,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377308060,"statfs":{"total":386547056640,"available":386363453440,"internally_reserved":0,"allocated":42668032,"data_stored":88427469,"data_compressed":4499818,"data_compressed_allocated":40599552,"data_compressed_original":87613440,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13295616,"num_objects":-203,"num_object_clones":0,"num_object_copies":-609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1623,"num_write_kb":-12984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001539"},"pg_stats":[{"pgid":"4.f","version":"51'224","reported_seq":111,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.104797+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:17.104797+0000","last_peered":"2024-04-22T01:59:17.104797+0000","last_clean":"2024-04-22T01:59:17.104797+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:17.104797+0000","last_undegraded":"2024-04-22T01:59:17.104797+0000","last_fullsized":"2024-04-22T01:59:17.104797+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"51'264","reported_seq":217,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.088098+0000","last_change":"2024-04-22T01:58:49.104590+0000","last_active":"2024-04-22T01:58:48.082724+0000","last_peered":"2024-04-22T01:58:46.487436+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:13.125202+0000","last_became_peered":"2024-04-22T01:58:13.125202+0000","last_unstale":"2024-04-22T01:59:17.050290+0000","last_undegraded":"2024-04-22T01:58:49.088098+0000","last_fullsized":"2024-04-22T01:58:49.088098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":47,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.549027+0000","last_change":"2024-04-22T01:59:19.550741+0000","last_active":"2024-04-22T01:59:19.549027+0000","last_peered":"2024-04-22T01:59:19.549027+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.122641+0000","last_became_peered":"2024-04-22T01:59:19.122641+0000","last_unstale":"2024-04-22T01:59:19.549027+0000","last_undegraded":"2024-04-22T01:59:19.103724+0000","last_fullsized":"2024-04-22T01:59:19.101613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"51'288","reported_seq":359,"reported_epoch":63,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:18.147279+0000","last_change":"2024-04-22T01:59:18.146416+0000","last_active":"2024-04-22T01:59:18.147279+0000","last_peered":"2024-04-22T01:59:18.147279+0000","last_clean":"2024-04-22T01:59:00.086529+0000","last_became_active":"2024-04-22T01:59:18.146416+0000","last_became_peered":"2024-04-22T01:59:18.146416+0000","last_unstale":"2024-04-22T01:59:18.147279+0000","last_undegraded":"2024-04-22T01:59:18.075324+0000","last_fullsized":"2024-04-22T01:59:18.073778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":36}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":286,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.101558+0000","last_change":"2024-04-22T01:59:18.200119+0000","last_active":"2024-04-22T01:59:19.101558+0000","last_peered":"2024-04-22T01:59:19.101558+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:59:18.200119+0000","last_became_peered":"2024-04-22T01:59:18.200119+0000","last_unstale":"2024-04-22T01:59:19.101558+0000","last_undegraded":"2024-04-22T01:59:18.087681+0000","last_fullsized":"2024-04-22T01:59:18.086263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"51'192","reported_seq":5,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-22T01:59:18.105700+0000","last_change":"2024-04-22T01:59:18.105700+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:18.105700+0000","last_undegraded":"2024-04-22T01:59:18.105700+0000","last_fullsized":"2024-04-22T01:59:18.105700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"51'200","reported_seq":9,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.139804+0000","last_change":"2024-04-22T01:59:18.138497+0000","last_active":"2024-04-22T01:59:18.139804+0000","last_peered":"2024-04-22T01:59:18.139804+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.138497+0000","last_became_peered":"2024-04-22T01:59:18.138497+0000","last_unstale":"2024-04-22T01:59:18.139804+0000","last_undegraded":"2024-04-22T01:59:18.139804+0000","last_fullsized":"2024-04-22T01:59:18.100062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.083651+0000","last_change":"2024-04-22T01:59:18.160401+0000","last_active":"2024-04-22T01:59:19.083651+0000","last_peered":"2024-04-22T01:59:19.083651+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.160401+0000","last_became_peered":"2024-04-22T01:59:18.160401+0000","last_unstale":"2024-04-22T01:59:19.083651+0000","last_undegraded":"2024-04-22T01:59:19.083651+0000","last_fullsized":"2024-04-22T01:59:18.083620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.078848+0000","last_change":"2024-04-22T01:59:18.145130+0000","last_active":"2024-04-22T01:59:19.078848+0000","last_peered":"2024-04-22T01:59:19.078848+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.145130+0000","last_became_peered":"2024-04-22T01:59:18.145130+0000","last_unstale":"2024-04-22T01:59:19.078848+0000","last_undegraded":"2024-04-22T01:59:19.078848+0000","last_fullsized":"2024-04-22T01:59:18.073418+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":295,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:59:18.084023+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:18.084023+0000","last_peered":"2024-04-22T01:59:18.084023+0000","last_clean":"2024-04-22T01:59:18.084023+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:18.084023+0000","last_undegraded":"2024-04-22T01:59:18.084023+0000","last_fullsized":"2024-04-22T01:59:18.084023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.077390+0000","last_change":"2024-04-22T01:59:18.185653+0000","last_active":"2024-04-22T01:59:19.077390+0000","last_peered":"2024-04-22T01:59:19.077390+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:18.185653+0000","last_became_peered":"2024-04-22T01:59:18.185653+0000","last_unstale":"2024-04-22T01:59:19.077390+0000","last_undegraded":"2024-04-22T01:59:19.077390+0000","last_fullsized":"2024-04-22T01:59:18.080636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":360,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.102789+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:19.102789+0000","last_peered":"2024-04-22T01:59:19.102789+0000","last_clean":"2024-04-22T01:59:19.102789+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:19.102789+0000","last_undegraded":"2024-04-22T01:59:19.102789+0000","last_fullsized":"2024-04-22T01:59:19.102789+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.105337+0000","last_change":"2024-04-22T01:59:18.233671+0000","last_active":"2024-04-22T01:59:19.105337+0000","last_peered":"2024-04-22T01:59:19.105337+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:18.233671+0000","last_became_peered":"2024-04-22T01:59:18.233671+0000","last_unstale":"2024-04-22T01:59:19.105337+0000","last_undegraded":"2024-04-22T01:59:19.105337+0000","last_fullsized":"2024-04-22T01:59:18.079371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"51'168","reported_seq":151,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-04-22T01:59:00.172333+0000","last_change":"2024-04-22T01:58:49.376458+0000","last_active":"2024-04-22T01:59:00.172333+0000","last_peered":"2024-04-22T01:59:00.172333+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:58:49.372257+0000","last_became_peered":"2024-04-22T01:58:49.372257+0000","last_unstale":"2024-04-22T01:59:00.172333+0000","last_undegraded":"2024-04-22T01:58:49.079110+0000","last_fullsized":"2024-04-22T01:58:49.067926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":14}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.182127+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:18.182127+0000","last_peered":"2024-04-22T01:59:18.182127+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:18.182127+0000","last_undegraded":"2024-04-22T01:59:18.182127+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"51'272","reported_seq":6,"reported_epoch":54,"state":"stale+activating+undersized+remapped","last_fresh":"2024-04-22T01:58:49.108331+0000","last_change":"2024-04-22T01:58:49.124985+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:17.050314+0000","last_undegraded":"2024-04-22T01:58:49.108331+0000","last_fullsized":"2024-04-22T01:58:49.108331+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":-24}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.097977+0000","last_change":"2024-04-22T01:59:18.230079+0000","last_active":"2024-04-22T01:59:19.097977+0000","last_peered":"2024-04-22T01:59:19.097977+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:18.230079+0000","last_became_peered":"2024-04-22T01:59:18.230079+0000","last_unstale":"2024-04-22T01:59:19.097977+0000","last_undegraded":"2024-04-22T01:59:19.097977+0000","last_fullsized":"2024-04-22T01:59:18.101172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"51'200","reported_seq":5,"reported_epoch":63,"state":"peering","last_fresh":"2024-04-22T01:59:18.098361+0000","last_change":"2024-04-22T01:59:18.098361+0000","last_active":"2024-04-22T01:57:35.038204+0000","last_peered":"2024-04-22T01:57:35.038204+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-22T01:59:18.098361+0000","last_undegraded":"2024-04-22T01:59:18.098361+0000","last_fullsized":"2024-04-22T01:59:18.098361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":97,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.066184+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:17.066184+0000","last_peered":"2024-04-22T01:59:17.066184+0000","last_clean":"2024-04-22T01:59:17.066184+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:17.066184+0000","last_undegraded":"2024-04-22T01:59:17.066184+0000","last_fullsized":"2024-04-22T01:59:17.066184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"63'272","reported_seq":359,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.088085+0000","last_change":"2024-04-22T01:59:18.176827+0000","last_active":"2024-04-22T01:59:19.088085+0000","last_peered":"2024-04-22T01:59:19.088085+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:59:18.176827+0000","last_became_peered":"2024-04-22T01:59:18.176827+0000","last_unstale":"2024-04-22T01:59:19.088085+0000","last_undegraded":"2024-04-22T01:59:18.071711+0000","last_fullsized":"2024-04-22T01:59:18.070350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":34}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":159,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-04-22T01:59:18.105076+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:18.105076+0000","last_peered":"2024-04-22T01:59:18.105076+0000","last_clean":"2024-04-22T01:59:18.105076+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:18.105076+0000","last_undegraded":"2024-04-22T01:59:18.105076+0000","last_fullsized":"2024-04-22T01:59:18.105076+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":287,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.098286+0000","last_change":"2024-04-22T01:59:18.183201+0000","last_active":"2024-04-22T01:59:19.098286+0000","last_peered":"2024-04-22T01:59:19.098286+0000","last_clean":"2024-04-22T01:59:00.163168+0000","last_became_active":"2024-04-22T01:59:18.183201+0000","last_became_peered":"2024-04-22T01:59:18.183201+0000","last_unstale":"2024-04-22T01:59:19.098286+0000","last_undegraded":"2024-04-22T01:59:18.081510+0000","last_fullsized":"2024-04-22T01:59:18.080121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":97,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-04-22T01:59:17.103127+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:17.103127+0000","last_peered":"2024-04-22T01:59:17.103127+0000","last_clean":"2024-04-22T01:59:17.103127+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:17.103127+0000","last_undegraded":"2024-04-22T01:59:17.103127+0000","last_fullsized":"2024-04-22T01:59:17.103127+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"52'304","reported_seq":9,"reported_epoch":63,"state":"active+undersized","last_fresh":"2024-04-22T01:59:18.150784+0000","last_change":"2024-04-22T01:59:18.150038+0000","last_active":"2024-04-22T01:59:18.150784+0000","last_peered":"2024-04-22T01:59:18.150784+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.150038+0000","last_became_peered":"2024-04-22T01:59:18.150038+0000","last_unstale":"2024-04-22T01:59:18.150784+0000","last_undegraded":"2024-04-22T01:59:18.150784+0000","last_fullsized":"2024-04-22T01:59:18.108583+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.107531+0000","last_change":"2024-04-22T01:59:18.209616+0000","last_active":"2024-04-22T01:59:19.107531+0000","last_peered":"2024-04-22T01:59:19.107531+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:18.209616+0000","last_became_peered":"2024-04-22T01:59:18.209616+0000","last_unstale":"2024-04-22T01:59:19.107531+0000","last_undegraded":"2024-04-22T01:59:19.107531+0000","last_fullsized":"2024-04-22T01:59:18.094962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6266880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3792,"ondisk_log_size":3792,"up":37,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39740,"kb_used_data":12916,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94332100,"statfs":{"total":96636764160,"available":96596070400,"internally_reserved":0,"allocated":13225984,"data_stored":26084281,"data_compressed":1335325,"data_compressed_allocated":12689408,"data_compressed_original":25870336,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299422,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35044,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94336796,"statfs":{"total":96636764160,"available":96600879104,"internally_reserved":0,"allocated":3829760,"data_stored":12174265,"data_compressed":595591,"data_compressed_allocated":3293184,"data_compressed_original":11960320,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51832,"kb_used_data":12016,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":12304384,"data_stored":24272297,"data_compressed":1241201,"data_compressed_allocated":11751424,"data_compressed_original":24051712,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:21.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:22.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 475 objects 2024-04-22T01:59:22.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :2.08079 2024-04-22T01:59:23.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:24.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:24.103+0000 7fe75e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:24.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:24.108+0000 7fe75e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:24.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:24.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:24.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:24.880+0000 7ff08bd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:24.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:24.887+0000 7ff08bd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:25.070 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:25.071 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:25.083 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-22T01:59:24.235665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7930400,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1889,"num_write_kb":13790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4151,"ondisk_log_size":4151,"up":57,"acting":57,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":163940,"kb_used_data":26308,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377323420,"statfs":{"total":386547056640,"available":386379182080,"internally_reserved":0,"allocated":26939392,"data_stored":52985797,"data_compressed":2688034,"data_compressed_allocated":24834048,"data_compressed_original":52158464,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18145280,"num_objects":-277,"num_object_clones":0,"num_object_copies":-831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1359,"num_write_kb":-12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":3997696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.185419"},"pg_stats":[{"pgid":"4.f","version":"64'251","reported_seq":145,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:21.791008+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:21.791008+0000","last_peered":"2024-04-22T01:59:21.791008+0000","last_clean":"2024-04-22T01:59:21.791008+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:21.791008+0000","last_undegraded":"2024-04-22T01:59:21.791008+0000","last_fullsized":"2024-04-22T01:59:21.791008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"64'296","reported_seq":307,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.111439+0000","last_change":"2024-04-22T01:59:19.856201+0000","last_active":"2024-04-22T01:59:22.111439+0000","last_peered":"2024-04-22T01:59:22.111439+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.153724+0000","last_became_peered":"2024-04-22T01:59:19.153724+0000","last_unstale":"2024-04-22T01:59:22.111439+0000","last_undegraded":"2024-04-22T01:59:22.111439+0000","last_fullsized":"2024-04-22T01:59:19.092941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":47,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.549027+0000","last_change":"2024-04-22T01:59:19.550741+0000","last_active":"2024-04-22T01:59:19.549027+0000","last_peered":"2024-04-22T01:59:19.549027+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.122641+0000","last_became_peered":"2024-04-22T01:59:19.122641+0000","last_unstale":"2024-04-22T01:59:19.549027+0000","last_undegraded":"2024-04-22T01:59:19.103724+0000","last_fullsized":"2024-04-22T01:59:19.101613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"64'323","reported_seq":398,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:22.096740+0000","last_change":"2024-04-22T01:59:18.146416+0000","last_active":"2024-04-22T01:59:22.096740+0000","last_peered":"2024-04-22T01:59:22.096740+0000","last_clean":"2024-04-22T01:59:00.086529+0000","last_became_active":"2024-04-22T01:59:18.146416+0000","last_became_peered":"2024-04-22T01:59:18.146416+0000","last_unstale":"2024-04-22T01:59:22.096740+0000","last_undegraded":"2024-04-22T01:59:18.075324+0000","last_fullsized":"2024-04-22T01:59:18.073778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":286,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.101558+0000","last_change":"2024-04-22T01:59:18.200119+0000","last_active":"2024-04-22T01:59:19.101558+0000","last_peered":"2024-04-22T01:59:19.101558+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:59:18.200119+0000","last_became_peered":"2024-04-22T01:59:18.200119+0000","last_unstale":"2024-04-22T01:59:19.101558+0000","last_undegraded":"2024-04-22T01:59:18.087681+0000","last_fullsized":"2024-04-22T01:59:18.086263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"64'215","reported_seq":69,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.113986+0000","last_change":"2024-04-22T01:59:19.706646+0000","last_active":"2024-04-22T01:59:22.113986+0000","last_peered":"2024-04-22T01:59:22.113986+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.135796+0000","last_became_peered":"2024-04-22T01:59:19.135796+0000","last_unstale":"2024-04-22T01:59:22.113986+0000","last_undegraded":"2024-04-22T01:59:22.113986+0000","last_fullsized":"2024-04-22T01:59:19.103002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"64'224","reported_seq":37,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:21.957045+0000","last_change":"2024-04-22T01:59:18.138497+0000","last_active":"2024-04-22T01:59:21.957045+0000","last_peered":"2024-04-22T01:59:21.957045+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.138497+0000","last_became_peered":"2024-04-22T01:59:18.138497+0000","last_unstale":"2024-04-22T01:59:21.957045+0000","last_undegraded":"2024-04-22T01:59:21.957045+0000","last_fullsized":"2024-04-22T01:59:18.100062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.083651+0000","last_change":"2024-04-22T01:59:18.160401+0000","last_active":"2024-04-22T01:59:19.083651+0000","last_peered":"2024-04-22T01:59:19.083651+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.160401+0000","last_became_peered":"2024-04-22T01:59:18.160401+0000","last_unstale":"2024-04-22T01:59:19.083651+0000","last_undegraded":"2024-04-22T01:59:19.083651+0000","last_fullsized":"2024-04-22T01:59:18.083620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.078848+0000","last_change":"2024-04-22T01:59:18.145130+0000","last_active":"2024-04-22T01:59:19.078848+0000","last_peered":"2024-04-22T01:59:19.078848+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.145130+0000","last_became_peered":"2024-04-22T01:59:18.145130+0000","last_unstale":"2024-04-22T01:59:19.078848+0000","last_undegraded":"2024-04-22T01:59:19.078848+0000","last_fullsized":"2024-04-22T01:59:18.073418+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":298,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.085801+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:19.085801+0000","last_peered":"2024-04-22T01:59:19.085801+0000","last_clean":"2024-04-22T01:59:19.085801+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:19.085801+0000","last_undegraded":"2024-04-22T01:59:19.085801+0000","last_fullsized":"2024-04-22T01:59:19.085801+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.077390+0000","last_change":"2024-04-22T01:59:18.185653+0000","last_active":"2024-04-22T01:59:19.077390+0000","last_peered":"2024-04-22T01:59:19.077390+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:18.185653+0000","last_became_peered":"2024-04-22T01:59:18.185653+0000","last_unstale":"2024-04-22T01:59:19.077390+0000","last_undegraded":"2024-04-22T01:59:19.077390+0000","last_fullsized":"2024-04-22T01:59:18.080636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":360,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.102789+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:19.102789+0000","last_peered":"2024-04-22T01:59:19.102789+0000","last_clean":"2024-04-22T01:59:19.102789+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:19.102789+0000","last_undegraded":"2024-04-22T01:59:19.102789+0000","last_fullsized":"2024-04-22T01:59:19.102789+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.105337+0000","last_change":"2024-04-22T01:59:18.233671+0000","last_active":"2024-04-22T01:59:19.105337+0000","last_peered":"2024-04-22T01:59:19.105337+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:18.233671+0000","last_became_peered":"2024-04-22T01:59:18.233671+0000","last_unstale":"2024-04-22T01:59:19.105337+0000","last_undegraded":"2024-04-22T01:59:19.105337+0000","last_fullsized":"2024-04-22T01:59:18.079371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"64'195","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:22.160062+0000","last_change":"2024-04-22T01:59:19.148356+0000","last_active":"2024-04-22T01:59:22.160062+0000","last_peered":"2024-04-22T01:59:22.160062+0000","last_clean":"2024-04-22T01:59:22.160062+0000","last_became_active":"2024-04-22T01:59:19.142747+0000","last_became_peered":"2024-04-22T01:59:19.142747+0000","last_unstale":"2024-04-22T01:59:22.160062+0000","last_undegraded":"2024-04-22T01:59:22.160062+0000","last_fullsized":"2024-04-22T01:59:22.160062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":140,"num_write_kb":944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.080514+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:19.080514+0000","last_peered":"2024-04-22T01:59:19.080514+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:19.080514+0000","last_undegraded":"2024-04-22T01:59:19.080514+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"64'305","reported_seq":193,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.088814+0000","last_change":"2024-04-22T01:59:21.544586+0000","last_active":"2024-04-22T01:59:22.088814+0000","last_peered":"2024-04-22T01:59:22.088814+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.151632+0000","last_became_peered":"2024-04-22T01:59:19.151632+0000","last_unstale":"2024-04-22T01:59:22.088814+0000","last_undegraded":"2024-04-22T01:59:22.088814+0000","last_fullsized":"2024-04-22T01:59:19.095409+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.097977+0000","last_change":"2024-04-22T01:59:18.230079+0000","last_active":"2024-04-22T01:59:19.097977+0000","last_peered":"2024-04-22T01:59:19.097977+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:18.230079+0000","last_became_peered":"2024-04-22T01:59:18.230079+0000","last_unstale":"2024-04-22T01:59:19.097977+0000","last_undegraded":"2024-04-22T01:59:19.097977+0000","last_fullsized":"2024-04-22T01:59:18.101172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"64'224","reported_seq":68,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.082208+0000","last_change":"2024-04-22T01:59:19.661424+0000","last_active":"2024-04-22T01:59:22.082208+0000","last_peered":"2024-04-22T01:59:22.082208+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.143224+0000","last_became_peered":"2024-04-22T01:59:19.143224+0000","last_unstale":"2024-04-22T01:59:22.082208+0000","last_undegraded":"2024-04-22T01:59:22.082208+0000","last_fullsized":"2024-04-22T01:59:19.106890+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.077409+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:19.077409+0000","last_peered":"2024-04-22T01:59:19.077409+0000","last_clean":"2024-04-22T01:59:19.077409+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:19.077409+0000","last_undegraded":"2024-04-22T01:59:19.077409+0000","last_fullsized":"2024-04-22T01:59:19.077409+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"63'272","reported_seq":359,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.088085+0000","last_change":"2024-04-22T01:59:18.176827+0000","last_active":"2024-04-22T01:59:19.088085+0000","last_peered":"2024-04-22T01:59:19.088085+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:59:18.176827+0000","last_became_peered":"2024-04-22T01:59:18.176827+0000","last_unstale":"2024-04-22T01:59:19.088085+0000","last_undegraded":"2024-04-22T01:59:18.071711+0000","last_fullsized":"2024-04-22T01:59:18.070350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":34}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.095460+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:19.095460+0000","last_peered":"2024-04-22T01:59:19.095460+0000","last_clean":"2024-04-22T01:59:19.095460+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:19.095460+0000","last_undegraded":"2024-04-22T01:59:19.095460+0000","last_fullsized":"2024-04-22T01:59:19.095460+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":287,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.098286+0000","last_change":"2024-04-22T01:59:18.183201+0000","last_active":"2024-04-22T01:59:19.098286+0000","last_peered":"2024-04-22T01:59:19.098286+0000","last_clean":"2024-04-22T01:59:00.163168+0000","last_became_active":"2024-04-22T01:59:18.183201+0000","last_became_peered":"2024-04-22T01:59:18.183201+0000","last_unstale":"2024-04-22T01:59:19.098286+0000","last_undegraded":"2024-04-22T01:59:18.081510+0000","last_fullsized":"2024-04-22T01:59:18.080121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.089914+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:19.089914+0000","last_peered":"2024-04-22T01:59:19.089914+0000","last_clean":"2024-04-22T01:59:19.089914+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:19.089914+0000","last_undegraded":"2024-04-22T01:59:19.089914+0000","last_fullsized":"2024-04-22T01:59:19.089914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"64'341","reported_seq":50,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.167669+0000","last_change":"2024-04-22T01:59:18.150038+0000","last_active":"2024-04-22T01:59:22.167669+0000","last_peered":"2024-04-22T01:59:22.167669+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.150038+0000","last_became_peered":"2024-04-22T01:59:18.150038+0000","last_unstale":"2024-04-22T01:59:22.167669+0000","last_undegraded":"2024-04-22T01:59:22.167669+0000","last_fullsized":"2024-04-22T01:59:18.108583+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.107531+0000","last_change":"2024-04-22T01:59:18.209616+0000","last_active":"2024-04-22T01:59:19.107531+0000","last_peered":"2024-04-22T01:59:19.107531+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:18.209616+0000","last_became_peered":"2024-04-22T01:59:18.209616+0000","last_unstale":"2024-04-22T01:59:19.107531+0000","last_undegraded":"2024-04-22T01:59:19.107531+0000","last_fullsized":"2024-04-22T01:59:18.094962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1752,"num_write_kb":11896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4054,"ondisk_log_size":4054,"up":35,"acting":35,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":593920,"data_stored":794025,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299427,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32020,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":733184,"data_stored":2022849,"data_compressed":89165,"data_compressed_allocated":176128,"data_compressed_original":1802240,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51832,"kb_used_data":12016,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":12304384,"data_stored":24272297,"data_compressed":1241201,"data_compressed_allocated":11751424,"data_compressed_original":24051712,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:25.084 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T01:59:25.084 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:59:25.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:25.190+0000 7f36708ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:25.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:25.198+0000 7f36708ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:25.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:25.380 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:25.392 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-04-22T01:59:24.235665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7930400,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1889,"num_write_kb":13790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4151,"ondisk_log_size":4151,"up":57,"acting":57,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":163940,"kb_used_data":26308,"kb_used_omap":0,"kb_used_meta":137600,"kb_avail":377323420,"statfs":{"total":386547056640,"available":386379182080,"internally_reserved":0,"allocated":26939392,"data_stored":52985797,"data_compressed":2688034,"data_compressed_allocated":24834048,"data_compressed_original":52158464,"omap_allocated":0,"internal_metadata":140902400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18145280,"num_objects":-277,"num_object_clones":0,"num_object_copies":-831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1359,"num_write_kb":-12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":3997696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.185419"},"pg_stats":[{"pgid":"4.f","version":"64'251","reported_seq":145,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:21.791008+0000","last_change":"2024-04-22T01:58:52.097302+0000","last_active":"2024-04-22T01:59:21.791008+0000","last_peered":"2024-04-22T01:59:21.791008+0000","last_clean":"2024-04-22T01:59:21.791008+0000","last_became_active":"2024-04-22T01:58:52.092790+0000","last_became_peered":"2024-04-22T01:58:52.092790+0000","last_unstale":"2024-04-22T01:59:21.791008+0000","last_undegraded":"2024-04-22T01:59:21.791008+0000","last_fullsized":"2024-04-22T01:59:21.791008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"64'296","reported_seq":307,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.111439+0000","last_change":"2024-04-22T01:59:19.856201+0000","last_active":"2024-04-22T01:59:22.111439+0000","last_peered":"2024-04-22T01:59:22.111439+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.153724+0000","last_became_peered":"2024-04-22T01:59:19.153724+0000","last_unstale":"2024-04-22T01:59:22.111439+0000","last_undegraded":"2024-04-22T01:59:22.111439+0000","last_fullsized":"2024-04-22T01:59:19.092941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"51'232","reported_seq":47,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.549027+0000","last_change":"2024-04-22T01:59:19.550741+0000","last_active":"2024-04-22T01:59:19.549027+0000","last_peered":"2024-04-22T01:59:19.549027+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.122641+0000","last_became_peered":"2024-04-22T01:59:19.122641+0000","last_unstale":"2024-04-22T01:59:19.549027+0000","last_undegraded":"2024-04-22T01:59:19.103724+0000","last_fullsized":"2024-04-22T01:59:19.101613+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"64'323","reported_seq":398,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:22.096740+0000","last_change":"2024-04-22T01:59:18.146416+0000","last_active":"2024-04-22T01:59:22.096740+0000","last_peered":"2024-04-22T01:59:22.096740+0000","last_clean":"2024-04-22T01:59:00.086529+0000","last_became_active":"2024-04-22T01:59:18.146416+0000","last_became_peered":"2024-04-22T01:59:18.146416+0000","last_unstale":"2024-04-22T01:59:22.096740+0000","last_undegraded":"2024-04-22T01:59:18.075324+0000","last_fullsized":"2024-04-22T01:59:18.073778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":2304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"51'200","reported_seq":286,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.101558+0000","last_change":"2024-04-22T01:59:18.200119+0000","last_active":"2024-04-22T01:59:19.101558+0000","last_peered":"2024-04-22T01:59:19.101558+0000","last_clean":"2024-04-22T01:57:44.185878+0000","last_became_active":"2024-04-22T01:59:18.200119+0000","last_became_peered":"2024-04-22T01:59:18.200119+0000","last_unstale":"2024-04-22T01:59:19.101558+0000","last_undegraded":"2024-04-22T01:59:18.087681+0000","last_fullsized":"2024-04-22T01:59:18.086263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":25}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"64'215","reported_seq":69,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.113986+0000","last_change":"2024-04-22T01:59:19.706646+0000","last_active":"2024-04-22T01:59:22.113986+0000","last_peered":"2024-04-22T01:59:22.113986+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.135796+0000","last_became_peered":"2024-04-22T01:59:19.135796+0000","last_unstale":"2024-04-22T01:59:22.113986+0000","last_undegraded":"2024-04-22T01:59:22.113986+0000","last_fullsized":"2024-04-22T01:59:19.103002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"64'224","reported_seq":37,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:21.957045+0000","last_change":"2024-04-22T01:59:18.138497+0000","last_active":"2024-04-22T01:59:21.957045+0000","last_peered":"2024-04-22T01:59:21.957045+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.138497+0000","last_became_peered":"2024-04-22T01:59:18.138497+0000","last_unstale":"2024-04-22T01:59:21.957045+0000","last_undegraded":"2024-04-22T01:59:21.957045+0000","last_fullsized":"2024-04-22T01:59:18.100062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"51'304","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.083651+0000","last_change":"2024-04-22T01:59:18.160401+0000","last_active":"2024-04-22T01:59:19.083651+0000","last_peered":"2024-04-22T01:59:19.083651+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.160401+0000","last_became_peered":"2024-04-22T01:59:18.160401+0000","last_unstale":"2024-04-22T01:59:19.083651+0000","last_undegraded":"2024-04-22T01:59:19.083651+0000","last_fullsized":"2024-04-22T01:59:18.083620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"52'200","reported_seq":12,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.078848+0000","last_change":"2024-04-22T01:59:18.145130+0000","last_active":"2024-04-22T01:59:19.078848+0000","last_peered":"2024-04-22T01:59:19.078848+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.145130+0000","last_became_peered":"2024-04-22T01:59:18.145130+0000","last_unstale":"2024-04-22T01:59:19.078848+0000","last_undegraded":"2024-04-22T01:59:19.078848+0000","last_fullsized":"2024-04-22T01:59:18.073418+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":298,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.085801+0000","last_change":"2024-04-22T01:58:48.146679+0000","last_active":"2024-04-22T01:59:19.085801+0000","last_peered":"2024-04-22T01:59:19.085801+0000","last_clean":"2024-04-22T01:59:19.085801+0000","last_became_active":"2024-04-22T01:58:48.142129+0000","last_became_peered":"2024-04-22T01:58:48.142129+0000","last_unstale":"2024-04-22T01:59:19.085801+0000","last_undegraded":"2024-04-22T01:59:19.085801+0000","last_fullsized":"2024-04-22T01:59:19.085801+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.077390+0000","last_change":"2024-04-22T01:59:18.185653+0000","last_active":"2024-04-22T01:59:19.077390+0000","last_peered":"2024-04-22T01:59:19.077390+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:18.185653+0000","last_became_peered":"2024-04-22T01:59:18.185653+0000","last_unstale":"2024-04-22T01:59:19.077390+0000","last_undegraded":"2024-04-22T01:59:19.077390+0000","last_fullsized":"2024-04-22T01:59:18.080636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"51'248","reported_seq":360,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.102789+0000","last_change":"2024-04-22T01:58:51.134246+0000","last_active":"2024-04-22T01:59:19.102789+0000","last_peered":"2024-04-22T01:59:19.102789+0000","last_clean":"2024-04-22T01:59:19.102789+0000","last_became_active":"2024-04-22T01:58:51.129631+0000","last_became_peered":"2024-04-22T01:58:51.129631+0000","last_unstale":"2024-04-22T01:59:19.102789+0000","last_undegraded":"2024-04-22T01:59:19.102789+0000","last_fullsized":"2024-04-22T01:59:19.102789+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.105337+0000","last_change":"2024-04-22T01:59:18.233671+0000","last_active":"2024-04-22T01:59:19.105337+0000","last_peered":"2024-04-22T01:59:19.105337+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:18.233671+0000","last_became_peered":"2024-04-22T01:59:18.233671+0000","last_unstale":"2024-04-22T01:59:19.105337+0000","last_undegraded":"2024-04-22T01:59:19.105337+0000","last_fullsized":"2024-04-22T01:59:18.079371+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"64'195","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:22.160062+0000","last_change":"2024-04-22T01:59:19.148356+0000","last_active":"2024-04-22T01:59:22.160062+0000","last_peered":"2024-04-22T01:59:22.160062+0000","last_clean":"2024-04-22T01:59:22.160062+0000","last_became_active":"2024-04-22T01:59:19.142747+0000","last_became_peered":"2024-04-22T01:59:19.142747+0000","last_unstale":"2024-04-22T01:59:22.160062+0000","last_undegraded":"2024-04-22T01:59:22.160062+0000","last_fullsized":"2024-04-22T01:59:22.160062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":140,"num_write_kb":944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.080514+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:19.080514+0000","last_peered":"2024-04-22T01:59:19.080514+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:19.080514+0000","last_undegraded":"2024-04-22T01:59:19.080514+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"64'305","reported_seq":193,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.088814+0000","last_change":"2024-04-22T01:59:21.544586+0000","last_active":"2024-04-22T01:59:22.088814+0000","last_peered":"2024-04-22T01:59:22.088814+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.151632+0000","last_became_peered":"2024-04-22T01:59:19.151632+0000","last_unstale":"2024-04-22T01:59:22.088814+0000","last_undegraded":"2024-04-22T01:59:22.088814+0000","last_fullsized":"2024-04-22T01:59:19.095409+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":134,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.097977+0000","last_change":"2024-04-22T01:59:18.230079+0000","last_active":"2024-04-22T01:59:19.097977+0000","last_peered":"2024-04-22T01:59:19.097977+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:18.230079+0000","last_became_peered":"2024-04-22T01:59:18.230079+0000","last_unstale":"2024-04-22T01:59:19.097977+0000","last_undegraded":"2024-04-22T01:59:19.097977+0000","last_fullsized":"2024-04-22T01:59:18.101172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"64'224","reported_seq":68,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.082208+0000","last_change":"2024-04-22T01:59:19.661424+0000","last_active":"2024-04-22T01:59:22.082208+0000","last_peered":"2024-04-22T01:59:22.082208+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:19.143224+0000","last_became_peered":"2024-04-22T01:59:19.143224+0000","last_unstale":"2024-04-22T01:59:22.082208+0000","last_undegraded":"2024-04-22T01:59:22.082208+0000","last_fullsized":"2024-04-22T01:59:19.106890+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":103,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.077409+0000","last_change":"2024-04-22T01:58:48.184413+0000","last_active":"2024-04-22T01:59:19.077409+0000","last_peered":"2024-04-22T01:59:19.077409+0000","last_clean":"2024-04-22T01:59:19.077409+0000","last_became_active":"2024-04-22T01:58:48.173670+0000","last_became_peered":"2024-04-22T01:58:48.173670+0000","last_unstale":"2024-04-22T01:59:19.077409+0000","last_undegraded":"2024-04-22T01:59:19.077409+0000","last_fullsized":"2024-04-22T01:59:19.077409+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"63'272","reported_seq":359,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.088085+0000","last_change":"2024-04-22T01:59:18.176827+0000","last_active":"2024-04-22T01:59:19.088085+0000","last_peered":"2024-04-22T01:59:19.088085+0000","last_clean":"2024-04-22T01:57:44.216788+0000","last_became_active":"2024-04-22T01:59:18.176827+0000","last_became_peered":"2024-04-22T01:59:18.176827+0000","last_unstale":"2024-04-22T01:59:19.088085+0000","last_undegraded":"2024-04-22T01:59:18.071711+0000","last_fullsized":"2024-04-22T01:59:18.070350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":34}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.095460+0000","last_change":"2024-04-22T01:58:48.160146+0000","last_active":"2024-04-22T01:59:19.095460+0000","last_peered":"2024-04-22T01:59:19.095460+0000","last_clean":"2024-04-22T01:59:19.095460+0000","last_became_active":"2024-04-22T01:58:48.154218+0000","last_became_peered":"2024-04-22T01:58:48.154218+0000","last_unstale":"2024-04-22T01:59:19.095460+0000","last_undegraded":"2024-04-22T01:59:19.095460+0000","last_fullsized":"2024-04-22T01:59:19.095460+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"51'224","reported_seq":287,"reported_epoch":64,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:19.098286+0000","last_change":"2024-04-22T01:59:18.183201+0000","last_active":"2024-04-22T01:59:19.098286+0000","last_peered":"2024-04-22T01:59:19.098286+0000","last_clean":"2024-04-22T01:59:00.163168+0000","last_became_active":"2024-04-22T01:59:18.183201+0000","last_became_peered":"2024-04-22T01:59:18.183201+0000","last_unstale":"2024-04-22T01:59:19.098286+0000","last_undegraded":"2024-04-22T01:59:18.081510+0000","last_fullsized":"2024-04-22T01:59:18.080121+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":176,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":22}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":103,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-22T01:59:19.089914+0000","last_change":"2024-04-22T01:58:48.201136+0000","last_active":"2024-04-22T01:59:19.089914+0000","last_peered":"2024-04-22T01:59:19.089914+0000","last_clean":"2024-04-22T01:59:19.089914+0000","last_became_active":"2024-04-22T01:58:48.196709+0000","last_became_peered":"2024-04-22T01:58:48.196709+0000","last_unstale":"2024-04-22T01:59:19.089914+0000","last_undegraded":"2024-04-22T01:59:19.089914+0000","last_fullsized":"2024-04-22T01:59:19.089914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"64'341","reported_seq":50,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:22.167669+0000","last_change":"2024-04-22T01:59:18.150038+0000","last_active":"2024-04-22T01:59:22.167669+0000","last_peered":"2024-04-22T01:59:22.167669+0000","last_clean":"2024-04-22T01:57:35.038204+0000","last_became_active":"2024-04-22T01:59:18.150038+0000","last_became_peered":"2024-04-22T01:59:18.150038+0000","last_unstale":"2024-04-22T01:59:22.167669+0000","last_undegraded":"2024-04-22T01:59:22.167669+0000","last_fullsized":"2024-04-22T01:59:18.108583+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:35.038204+0000","last_clean_scrub_stamp":"2024-04-22T01:57:35.038204+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":117,"reported_epoch":64,"state":"active+undersized","last_fresh":"2024-04-22T01:59:19.107531+0000","last_change":"2024-04-22T01:59:18.209616+0000","last_active":"2024-04-22T01:59:19.107531+0000","last_peered":"2024-04-22T01:59:19.107531+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:18.209616+0000","last_became_peered":"2024-04-22T01:59:18.209616+0000","last_unstale":"2024-04-22T01:59:19.107531+0000","last_undegraded":"2024-04-22T01:59:19.107531+0000","last_fullsized":"2024-04-22T01:59:18.094962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1752,"num_write_kb":11896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4054,"ondisk_log_size":4054,"up":35,"acting":35,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":593920,"data_stored":794025,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299427,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32020,"kb_used_data":716,"kb_used_omap":0,"kb_used_meta":31296,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":733184,"data_stored":2022849,"data_compressed":89165,"data_compressed_allocated":176128,"data_compressed_original":1802240,"omap_allocated":0,"internal_metadata":32047104},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":50,"seq":214748364850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51832,"kb_used_data":12016,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":12304384,"data_stored":24272297,"data_compressed":1241201,"data_compressed_allocated":11751424,"data_compressed_original":24051712,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:25.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 4.b 4.3 4.0 2024-04-22T01:59:25.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:25.496+0000 7f2c3995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:25.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:25.507+0000 7f2c3995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:25.689 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg(s) [4.0,4.3,4.b] on osd.0 to force-backfill; 2024-04-22T01:59:25.689 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T01:59:26.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:26.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:26.299+0000 7f694855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:26.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:26.304+0000 7f694855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:26.327 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:26.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:26.892 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-22T01:59:26.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-22T01:59:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:26.997+0000 7f7b9a2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:27.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:27.009+0000 7f7b9a2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:27.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2024-04-22T01:59:28.135 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_1' removed 2024-04-22T01:59:28.152 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-22T01:59:28.152 INFO:tasks.radosbench:Beginning radosbench... 2024-04-22T01:59:28.152 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-22T01:59:28.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_2 16 2024-04-22T01:59:28.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:28.286+0000 7fb928624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:28.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:28.299+0000 7fb928624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:28.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:28.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:28.481+0000 7fe1d1154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:28.486+0000 7fe1d1154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:28.511 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:28.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:29.139 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_2' created 2024-04-22T01:59:29.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-22T01:59:29.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:29.281+0000 7fe0b32ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:29.295+0000 7fe0b32ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:30.145 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-22T01:59:30.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:30.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:30.660+0000 7f3d62a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:30.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:30.666+0000 7f3d62a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:30.703 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 1, 0, 2] 2024-04-22T01:59:30.703 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:59:30.703 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [3, 1, 0, 2] 2024-04-22T01:59:30.703 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:59:30.704 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:59:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:30.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:30.780 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:59:31.164 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-22T01:59:31.166 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-22T01:59:31.166 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-22T01:59:31.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-04-22T01:59:31.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-22T01:59:31.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_35980 2024-04-22T01:59:31.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:59:31.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-04-22T01:59:32.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 24 8 0.062495 0.0625 0.13372 0.0956398 2024-04-22T01:59:32.583 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~39s 2024-04-22T01:59:32.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:32.849+0000 7fbaca66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:32.854+0000 7fbaca66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:33.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 24 8 0.0312466 0 - 0.0956398 2024-04-22T01:59:34.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 24 8 0.0208311 0 - 0.0956398 2024-04-22T01:59:35.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 24 8 0.0156232 0 - 0.0956398 2024-04-22T01:59:35.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [3, 1, 2] 2024-04-22T01:59:35.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:59:35.781 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-04-22T01:59:35.781 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:59:35.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:59:35.784 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:59:35.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:59:35.892 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:59:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:59:35.896 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:59:36.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 24 8 0.0124985 0 - 0.0956398 2024-04-22T01:59:37.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 24 8 0.0104155 0 - 0.0956398 2024-04-22T01:59:38.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~44s 2024-04-22T01:59:38.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 24 8 0.00892755 0 - 0.0956398 2024-04-22T01:59:39.230 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 24 8 0.00781164 0 - 0.0956398 2024-04-22T01:59:40.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 24 8 0.00694367 0 - 0.0956398 2024-04-22T01:59:40.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:59:40.994 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T01:59:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T01:59:40.998 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-04-22T01:59:41.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 24 8 0.00624933 0 - 0.0956398 2024-04-22T01:59:42.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 24 8 0.00568122 0 - 0.0956398 2024-04-22T01:59:43.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 24 8 0.00520779 0 - 0.0956398 2024-04-22T01:59:43.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-04-22T01:59:44.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 24 8 0.0048072 0 - 0.0956398 2024-04-22T01:59:45.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 24 8 0.00446383 0 - 0.0956398 2024-04-22T01:59:45.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-22T01:59:46.110 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-04-22T01:59:46.111 INFO:teuthology.orchestra.run.smithi107.stdout: "ops_in_flight": [] 2024-04-22T01:59:46.111 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-04-22T01:59:46.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 24 8 0.00416625 0 - 0.0956398 2024-04-22T01:59:47.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 214 198 0.0966701 0.0989583 0.0227047 1.28904 2024-04-22T01:59:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:48.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:48.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 378 362 0.166344 1.28125 0.0948655 0.749529 2024-04-22T01:59:49.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-04-22T01:59:49.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 559 543 0.235654 1.41406 0.104313 0.529086 2024-04-22T01:59:50.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:50.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:50.132+0000 7f173019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:50.138+0000 7f173019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:50.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 734 718 0.295201 1.36719 0.119262 0.421496 2024-04-22T01:59:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:50.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:51.121 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 1, 2, 0] 2024-04-22T01:59:51.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-22T01:59:51.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T01:59:51.233575+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.368676 2024-04-22T01:59:51.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T01:59:51.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 881 865 0.337858 1.14844 0.210086 0.368676 2024-04-22T01:59:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:51.245+0000 7fdc2b62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:51.257+0000 7fdc2b62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:52.197 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-04-22T01:59:52.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-22T01:59:52.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 1054 1038 0.386123 1.35156 0.0926617 0.322647 2024-04-22T01:59:52.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:52.336+0000 7fbe4d2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:52.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:52.347+0000 7fbe4d2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:52.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:52.472+0000 7fda35c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:52.479+0000 7fda35c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:52.546 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:52.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:53.204 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-04-22T01:59:53.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-22T01:59:53.231 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 1207 1191 0.422899 1.19531 0.0802631 0.29432 2024-04-22T01:59:53.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:53.353+0000 7fa074ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:53.363+0000 7fa074ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:54.205 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-04-22T01:59:54.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-22T01:59:54.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 1356 1340 0.455119 1.16406 0.11827 0.274114 2024-04-22T01:59:54.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:54.351+0000 7f8b67224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:54.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:54.362+0000 7f8b67224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:54.600 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~60s 2024-04-22T01:59:54.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:54.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:54.736+0000 7f32bd6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:54.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:54.742+0000 7f32bd6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:54.771 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:54.837 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:55.210 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-04-22T01:59:55.224 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-22T01:59:55.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:55.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 1512 1496 0.486933 1.21875 0.15681 0.255984 2024-04-22T01:59:55.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:55.341+0000 7fa6484ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:55.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:55.351+0000 7fa6484ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:55.622 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:55.640 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-04-22T01:59:55.214943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10232352,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1314,"num_write_kb":11310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1266,"ondisk_log_size":1266,"up":60,"acting":60,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":158744,"kb_used_data":25144,"kb_used_omap":0,"kb_used_meta":133568,"kb_avail":377328616,"statfs":{"total":386547056640,"available":386384502784,"internally_reserved":0,"allocated":25747456,"data_stored":49154949,"data_compressed":2487303,"data_compressed_allocated":23371776,"data_compressed_original":48218112,"omap_allocated":0,"internal_metadata":136773632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6463488,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":6312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.972646"},"pg_stats":[{"pgid":"5.f","version":"74'56","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249692+0000","last_change":"2024-04-22T01:59:45.265202+0000","last_active":"2024-04-22T01:59:52.249692+0000","last_peered":"2024-04-22T01:59:52.249692+0000","last_clean":"2024-04-22T01:59:52.249692+0000","last_became_active":"2024-04-22T01:59:45.259408+0000","last_became_peered":"2024-04-22T01:59:45.259408+0000","last_unstale":"2024-04-22T01:59:52.249692+0000","last_undegraded":"2024-04-22T01:59:52.249692+0000","last_fullsized":"2024-04-22T01:59:52.249692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"76'59","reported_seq":87,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.797741+0000","last_change":"2024-04-22T01:59:45.221309+0000","last_active":"2024-04-22T01:59:53.797741+0000","last_peered":"2024-04-22T01:59:53.797741+0000","last_clean":"2024-04-22T01:59:53.797741+0000","last_became_active":"2024-04-22T01:59:45.213752+0000","last_became_peered":"2024-04-22T01:59:45.213752+0000","last_unstale":"2024-04-22T01:59:53.797741+0000","last_undegraded":"2024-04-22T01:59:53.797741+0000","last_fullsized":"2024-04-22T01:59:53.797741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"75'112","reported_seq":139,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.291542+0000","last_change":"2024-04-22T01:59:45.200056+0000","last_active":"2024-04-22T01:59:53.291542+0000","last_peered":"2024-04-22T01:59:53.291542+0000","last_clean":"2024-04-22T01:59:53.291542+0000","last_became_active":"2024-04-22T01:59:45.188317+0000","last_became_peered":"2024-04-22T01:59:45.188317+0000","last_unstale":"2024-04-22T01:59:53.291542+0000","last_undegraded":"2024-04-22T01:59:53.291542+0000","last_fullsized":"2024-04-22T01:59:53.291542+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"75'92","reported_seq":124,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.176755+0000","last_change":"2024-04-22T01:59:31.285251+0000","last_active":"2024-04-22T01:59:53.176755+0000","last_peered":"2024-04-22T01:59:53.176755+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.253372+0000","last_became_peered":"2024-04-22T01:59:30.253372+0000","last_unstale":"2024-04-22T01:59:53.176755+0000","last_undegraded":"2024-04-22T01:59:31.273058+0000","last_fullsized":"2024-04-22T01:59:30.169001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":761856,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"75'136","reported_seq":158,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.295501+0000","last_change":"2024-04-22T01:59:46.301600+0000","last_active":"2024-04-22T01:59:53.295501+0000","last_peered":"2024-04-22T01:59:53.295501+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.153369+0000","last_became_peered":"2024-04-22T01:59:45.153369+0000","last_unstale":"2024-04-22T01:59:53.295501+0000","last_undegraded":"2024-04-22T01:59:46.288211+0000","last_fullsized":"2024-04-22T01:59:45.081008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":17}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"74'48","reported_seq":82,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.252166+0000","last_change":"2024-04-22T01:59:45.247767+0000","last_active":"2024-04-22T01:59:52.252166+0000","last_peered":"2024-04-22T01:59:52.252166+0000","last_clean":"2024-04-22T01:59:52.252166+0000","last_became_active":"2024-04-22T01:59:45.242401+0000","last_became_peered":"2024-04-22T01:59:45.242401+0000","last_unstale":"2024-04-22T01:59:52.252166+0000","last_undegraded":"2024-04-22T01:59:52.252166+0000","last_fullsized":"2024-04-22T01:59:52.252166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.318994+0000","last_change":"2024-04-22T01:59:49.041718+0000","last_active":"2024-04-22T01:59:53.318994+0000","last_peered":"2024-04-22T01:59:53.318994+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.207850+0000","last_became_peered":"2024-04-22T01:59:45.207850+0000","last_unstale":"2024-04-22T01:59:53.318994+0000","last_undegraded":"2024-04-22T01:59:49.025876+0000","last_fullsized":"2024-04-22T01:59:45.092045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"75'40","reported_seq":62,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.274559+0000","last_change":"2024-04-22T01:59:47.695363+0000","last_active":"2024-04-22T01:59:53.274559+0000","last_peered":"2024-04-22T01:59:53.274559+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.155260+0000","last_became_peered":"2024-04-22T01:59:45.155260+0000","last_unstale":"2024-04-22T01:59:53.274559+0000","last_undegraded":"2024-04-22T01:59:47.678787+0000","last_fullsized":"2024-04-22T01:59:45.058562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"75'47","reported_seq":78,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.362923+0000","last_change":"2024-04-22T01:59:46.595867+0000","last_active":"2024-04-22T01:59:52.362923+0000","last_peered":"2024-04-22T01:59:52.362923+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.198798+0000","last_became_peered":"2024-04-22T01:59:45.198798+0000","last_unstale":"2024-04-22T01:59:52.362923+0000","last_undegraded":"2024-04-22T01:59:46.578328+0000","last_fullsized":"2024-04-22T01:59:45.066794+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":332,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.288619+0000","last_change":"2024-04-22T01:59:45.161116+0000","last_active":"2024-04-22T01:59:52.288619+0000","last_peered":"2024-04-22T01:59:52.288619+0000","last_clean":"2024-04-22T01:59:52.288619+0000","last_became_active":"2024-04-22T01:59:45.150739+0000","last_became_peered":"2024-04-22T01:59:45.150739+0000","last_unstale":"2024-04-22T01:59:52.288619+0000","last_undegraded":"2024-04-22T01:59:52.288619+0000","last_fullsized":"2024-04-22T01:59:52.288619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.278096+0000","last_change":"2024-04-22T01:59:45.165595+0000","last_active":"2024-04-22T01:59:53.278096+0000","last_peered":"2024-04-22T01:59:53.278096+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:45.165595+0000","last_became_peered":"2024-04-22T01:59:45.165595+0000","last_unstale":"2024-04-22T01:59:53.278096+0000","last_undegraded":"2024-04-22T01:59:53.278096+0000","last_fullsized":"2024-04-22T01:59:45.065478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"74'80","reported_seq":102,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.283368+0000","last_change":"2024-04-22T01:59:47.025432+0000","last_active":"2024-04-22T01:59:53.283368+0000","last_peered":"2024-04-22T01:59:53.283368+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.183869+0000","last_became_peered":"2024-04-22T01:59:45.183869+0000","last_unstale":"2024-04-22T01:59:53.283368+0000","last_undegraded":"2024-04-22T01:59:47.005374+0000","last_fullsized":"2024-04-22T01:59:45.076865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.297858+0000","last_change":"2024-04-22T01:59:45.198972+0000","last_active":"2024-04-22T01:59:53.297858+0000","last_peered":"2024-04-22T01:59:53.297858+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:45.198972+0000","last_became_peered":"2024-04-22T01:59:45.198972+0000","last_unstale":"2024-04-22T01:59:53.297858+0000","last_undegraded":"2024-04-22T01:59:53.297858+0000","last_fullsized":"2024-04-22T01:59:45.072137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"74'80","reported_seq":114,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249333+0000","last_change":"2024-04-22T01:59:45.214443+0000","last_active":"2024-04-22T01:59:52.249333+0000","last_peered":"2024-04-22T01:59:52.249333+0000","last_clean":"2024-04-22T01:59:52.249333+0000","last_became_active":"2024-04-22T01:59:45.205299+0000","last_became_peered":"2024-04-22T01:59:45.205299+0000","last_unstale":"2024-04-22T01:59:52.249333+0000","last_undegraded":"2024-04-22T01:59:52.249333+0000","last_fullsized":"2024-04-22T01:59:52.249333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":125,"reported_epoch":75,"state":"active+undersized","last_fresh":"2024-04-22T01:59:52.284938+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:52.284938+0000","last_peered":"2024-04-22T01:59:52.284938+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:52.284938+0000","last_undegraded":"2024-04-22T01:59:52.284938+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"75'55","reported_seq":87,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.440956+0000","last_change":"2024-04-22T01:59:46.335573+0000","last_active":"2024-04-22T01:59:52.440956+0000","last_peered":"2024-04-22T01:59:52.440956+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.255441+0000","last_became_peered":"2024-04-22T01:59:30.255441+0000","last_unstale":"2024-04-22T01:59:52.440956+0000","last_undegraded":"2024-04-22T01:59:46.315293+0000","last_fullsized":"2024-04-22T01:59:30.161229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.307278+0000","last_change":"2024-04-22T01:59:45.189059+0000","last_active":"2024-04-22T01:59:53.307278+0000","last_peered":"2024-04-22T01:59:53.307278+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:45.189059+0000","last_became_peered":"2024-04-22T01:59:45.189059+0000","last_unstale":"2024-04-22T01:59:53.307278+0000","last_undegraded":"2024-04-22T01:59:53.307278+0000","last_fullsized":"2024-04-22T01:59:45.083825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"74'80","reported_seq":111,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.238498+0000","last_change":"2024-04-22T01:59:46.737480+0000","last_active":"2024-04-22T01:59:52.238498+0000","last_peered":"2024-04-22T01:59:52.238498+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.236750+0000","last_became_peered":"2024-04-22T01:59:30.236750+0000","last_unstale":"2024-04-22T01:59:52.238498+0000","last_undegraded":"2024-04-22T01:59:46.716626+0000","last_fullsized":"2024-04-22T01:59:30.165443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.254212+0000","last_change":"2024-04-22T01:59:45.223952+0000","last_active":"2024-04-22T01:59:52.254212+0000","last_peered":"2024-04-22T01:59:52.254212+0000","last_clean":"2024-04-22T01:59:52.254212+0000","last_became_active":"2024-04-22T01:59:45.217864+0000","last_became_peered":"2024-04-22T01:59:45.217864+0000","last_unstale":"2024-04-22T01:59:52.254212+0000","last_undegraded":"2024-04-22T01:59:52.254212+0000","last_fullsized":"2024-04-22T01:59:52.254212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.586623+0000","last_change":"2024-04-22T01:59:46.359994+0000","last_active":"2024-04-22T01:59:53.586623+0000","last_peered":"2024-04-22T01:59:53.586623+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.149551+0000","last_became_peered":"2024-04-22T01:59:45.149551+0000","last_unstale":"2024-04-22T01:59:53.586623+0000","last_undegraded":"2024-04-22T01:59:46.345979+0000","last_fullsized":"2024-04-22T01:59:45.075218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":196,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.295356+0000","last_change":"2024-04-22T01:59:45.178441+0000","last_active":"2024-04-22T01:59:52.295356+0000","last_peered":"2024-04-22T01:59:52.295356+0000","last_clean":"2024-04-22T01:59:52.295356+0000","last_became_active":"2024-04-22T01:59:45.171797+0000","last_became_peered":"2024-04-22T01:59:45.171797+0000","last_unstale":"2024-04-22T01:59:52.295356+0000","last_undegraded":"2024-04-22T01:59:52.295356+0000","last_fullsized":"2024-04-22T01:59:52.295356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"75'39","reported_seq":74,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.100195+0000","last_change":"2024-04-22T01:59:45.165903+0000","last_active":"2024-04-22T01:59:53.100195+0000","last_peered":"2024-04-22T01:59:53.100195+0000","last_clean":"2024-04-22T01:59:53.100195+0000","last_became_active":"2024-04-22T01:59:45.156195+0000","last_became_peered":"2024-04-22T01:59:45.156195+0000","last_unstale":"2024-04-22T01:59:53.100195+0000","last_undegraded":"2024-04-22T01:59:53.100195+0000","last_fullsized":"2024-04-22T01:59:53.100195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.241805+0000","last_change":"2024-04-22T01:59:45.241888+0000","last_active":"2024-04-22T01:59:52.241805+0000","last_peered":"2024-04-22T01:59:52.241805+0000","last_clean":"2024-04-22T01:59:52.241805+0000","last_became_active":"2024-04-22T01:59:45.236660+0000","last_became_peered":"2024-04-22T01:59:45.236660+0000","last_unstale":"2024-04-22T01:59:52.241805+0000","last_undegraded":"2024-04-22T01:59:52.241805+0000","last_fullsized":"2024-04-22T01:59:52.241805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"76'103","reported_seq":126,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.667754+0000","last_change":"2024-04-22T01:59:46.294155+0000","last_active":"2024-04-22T01:59:53.667754+0000","last_peered":"2024-04-22T01:59:53.667754+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.194515+0000","last_became_peered":"2024-04-22T01:59:45.194515+0000","last_unstale":"2024-04-22T01:59:53.667754+0000","last_undegraded":"2024-04-22T01:59:46.287653+0000","last_fullsized":"2024-04-22T01:59:45.080203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.294010+0000","last_change":"2024-04-22T01:59:45.203729+0000","last_active":"2024-04-22T01:59:53.294010+0000","last_peered":"2024-04-22T01:59:53.294010+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:45.203729+0000","last_became_peered":"2024-04-22T01:59:45.203729+0000","last_unstale":"2024-04-22T01:59:53.294010+0000","last_undegraded":"2024-04-22T01:59:53.294010+0000","last_fullsized":"2024-04-22T01:59:45.078142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":9641984,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":9416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1169,"ondisk_log_size":1169,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34740,"kb_used_data":3756,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94337100,"statfs":{"total":96636764160,"available":96601190400,"internally_reserved":0,"allocated":3846144,"data_stored":7147648,"data_compressed":355268,"data_compressed_allocated":3198976,"data_compressed_original":6889472,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39484,"kb_used_data":4020,"kb_used_omap":0,"kb_used_meta":35456,"kb_avail":94332356,"statfs":{"total":96636764160,"available":96596332544,"internally_reserved":0,"allocated":4116480,"data_stored":7701459,"data_compressed":383962,"data_compressed_allocated":3477504,"data_compressed_original":7446528,"omap_allocated":0,"internal_metadata":36306944},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":72,"seq":309237645321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31836,"kb_used_data":4372,"kb_used_omap":0,"kb_used_meta":27456,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":4476928,"data_stored":8409216,"data_compressed":420372,"data_compressed_allocated":3829760,"data_compressed_original":8151040,"omap_allocated":0,"internal_metadata":28114944},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:55.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:55.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:55.756+0000 7f562f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:55.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:55.765+0000 7f562f62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:56.042 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:56.042 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:56.058 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-04-22T01:59:55.214943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10232352,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1314,"num_write_kb":11310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1266,"ondisk_log_size":1266,"up":60,"acting":60,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":158744,"kb_used_data":25144,"kb_used_omap":0,"kb_used_meta":133568,"kb_avail":377328616,"statfs":{"total":386547056640,"available":386384502784,"internally_reserved":0,"allocated":25747456,"data_stored":49154949,"data_compressed":2487303,"data_compressed_allocated":23371776,"data_compressed_original":48218112,"omap_allocated":0,"internal_metadata":136773632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6463488,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":6312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.972646"},"pg_stats":[{"pgid":"5.f","version":"74'56","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249692+0000","last_change":"2024-04-22T01:59:45.265202+0000","last_active":"2024-04-22T01:59:52.249692+0000","last_peered":"2024-04-22T01:59:52.249692+0000","last_clean":"2024-04-22T01:59:52.249692+0000","last_became_active":"2024-04-22T01:59:45.259408+0000","last_became_peered":"2024-04-22T01:59:45.259408+0000","last_unstale":"2024-04-22T01:59:52.249692+0000","last_undegraded":"2024-04-22T01:59:52.249692+0000","last_fullsized":"2024-04-22T01:59:52.249692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"76'59","reported_seq":87,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.797741+0000","last_change":"2024-04-22T01:59:45.221309+0000","last_active":"2024-04-22T01:59:53.797741+0000","last_peered":"2024-04-22T01:59:53.797741+0000","last_clean":"2024-04-22T01:59:53.797741+0000","last_became_active":"2024-04-22T01:59:45.213752+0000","last_became_peered":"2024-04-22T01:59:45.213752+0000","last_unstale":"2024-04-22T01:59:53.797741+0000","last_undegraded":"2024-04-22T01:59:53.797741+0000","last_fullsized":"2024-04-22T01:59:53.797741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"75'112","reported_seq":139,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.291542+0000","last_change":"2024-04-22T01:59:45.200056+0000","last_active":"2024-04-22T01:59:53.291542+0000","last_peered":"2024-04-22T01:59:53.291542+0000","last_clean":"2024-04-22T01:59:53.291542+0000","last_became_active":"2024-04-22T01:59:45.188317+0000","last_became_peered":"2024-04-22T01:59:45.188317+0000","last_unstale":"2024-04-22T01:59:53.291542+0000","last_undegraded":"2024-04-22T01:59:53.291542+0000","last_fullsized":"2024-04-22T01:59:53.291542+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"75'92","reported_seq":124,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.176755+0000","last_change":"2024-04-22T01:59:31.285251+0000","last_active":"2024-04-22T01:59:53.176755+0000","last_peered":"2024-04-22T01:59:53.176755+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.253372+0000","last_became_peered":"2024-04-22T01:59:30.253372+0000","last_unstale":"2024-04-22T01:59:53.176755+0000","last_undegraded":"2024-04-22T01:59:31.273058+0000","last_fullsized":"2024-04-22T01:59:30.169001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":761856,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"75'136","reported_seq":158,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.295501+0000","last_change":"2024-04-22T01:59:46.301600+0000","last_active":"2024-04-22T01:59:53.295501+0000","last_peered":"2024-04-22T01:59:53.295501+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.153369+0000","last_became_peered":"2024-04-22T01:59:45.153369+0000","last_unstale":"2024-04-22T01:59:53.295501+0000","last_undegraded":"2024-04-22T01:59:46.288211+0000","last_fullsized":"2024-04-22T01:59:45.081008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":17}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"74'48","reported_seq":82,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.252166+0000","last_change":"2024-04-22T01:59:45.247767+0000","last_active":"2024-04-22T01:59:52.252166+0000","last_peered":"2024-04-22T01:59:52.252166+0000","last_clean":"2024-04-22T01:59:52.252166+0000","last_became_active":"2024-04-22T01:59:45.242401+0000","last_became_peered":"2024-04-22T01:59:45.242401+0000","last_unstale":"2024-04-22T01:59:52.252166+0000","last_undegraded":"2024-04-22T01:59:52.252166+0000","last_fullsized":"2024-04-22T01:59:52.252166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.318994+0000","last_change":"2024-04-22T01:59:49.041718+0000","last_active":"2024-04-22T01:59:53.318994+0000","last_peered":"2024-04-22T01:59:53.318994+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.207850+0000","last_became_peered":"2024-04-22T01:59:45.207850+0000","last_unstale":"2024-04-22T01:59:53.318994+0000","last_undegraded":"2024-04-22T01:59:49.025876+0000","last_fullsized":"2024-04-22T01:59:45.092045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"75'40","reported_seq":62,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.274559+0000","last_change":"2024-04-22T01:59:47.695363+0000","last_active":"2024-04-22T01:59:53.274559+0000","last_peered":"2024-04-22T01:59:53.274559+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.155260+0000","last_became_peered":"2024-04-22T01:59:45.155260+0000","last_unstale":"2024-04-22T01:59:53.274559+0000","last_undegraded":"2024-04-22T01:59:47.678787+0000","last_fullsized":"2024-04-22T01:59:45.058562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"75'47","reported_seq":78,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.362923+0000","last_change":"2024-04-22T01:59:46.595867+0000","last_active":"2024-04-22T01:59:52.362923+0000","last_peered":"2024-04-22T01:59:52.362923+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.198798+0000","last_became_peered":"2024-04-22T01:59:45.198798+0000","last_unstale":"2024-04-22T01:59:52.362923+0000","last_undegraded":"2024-04-22T01:59:46.578328+0000","last_fullsized":"2024-04-22T01:59:45.066794+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":332,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.288619+0000","last_change":"2024-04-22T01:59:45.161116+0000","last_active":"2024-04-22T01:59:52.288619+0000","last_peered":"2024-04-22T01:59:52.288619+0000","last_clean":"2024-04-22T01:59:52.288619+0000","last_became_active":"2024-04-22T01:59:45.150739+0000","last_became_peered":"2024-04-22T01:59:45.150739+0000","last_unstale":"2024-04-22T01:59:52.288619+0000","last_undegraded":"2024-04-22T01:59:52.288619+0000","last_fullsized":"2024-04-22T01:59:52.288619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.278096+0000","last_change":"2024-04-22T01:59:45.165595+0000","last_active":"2024-04-22T01:59:53.278096+0000","last_peered":"2024-04-22T01:59:53.278096+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:45.165595+0000","last_became_peered":"2024-04-22T01:59:45.165595+0000","last_unstale":"2024-04-22T01:59:53.278096+0000","last_undegraded":"2024-04-22T01:59:53.278096+0000","last_fullsized":"2024-04-22T01:59:45.065478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"74'80","reported_seq":102,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.283368+0000","last_change":"2024-04-22T01:59:47.025432+0000","last_active":"2024-04-22T01:59:53.283368+0000","last_peered":"2024-04-22T01:59:53.283368+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.183869+0000","last_became_peered":"2024-04-22T01:59:45.183869+0000","last_unstale":"2024-04-22T01:59:53.283368+0000","last_undegraded":"2024-04-22T01:59:47.005374+0000","last_fullsized":"2024-04-22T01:59:45.076865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.297858+0000","last_change":"2024-04-22T01:59:45.198972+0000","last_active":"2024-04-22T01:59:53.297858+0000","last_peered":"2024-04-22T01:59:53.297858+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:45.198972+0000","last_became_peered":"2024-04-22T01:59:45.198972+0000","last_unstale":"2024-04-22T01:59:53.297858+0000","last_undegraded":"2024-04-22T01:59:53.297858+0000","last_fullsized":"2024-04-22T01:59:45.072137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"74'80","reported_seq":114,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249333+0000","last_change":"2024-04-22T01:59:45.214443+0000","last_active":"2024-04-22T01:59:52.249333+0000","last_peered":"2024-04-22T01:59:52.249333+0000","last_clean":"2024-04-22T01:59:52.249333+0000","last_became_active":"2024-04-22T01:59:45.205299+0000","last_became_peered":"2024-04-22T01:59:45.205299+0000","last_unstale":"2024-04-22T01:59:52.249333+0000","last_undegraded":"2024-04-22T01:59:52.249333+0000","last_fullsized":"2024-04-22T01:59:52.249333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":125,"reported_epoch":75,"state":"active+undersized","last_fresh":"2024-04-22T01:59:52.284938+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:52.284938+0000","last_peered":"2024-04-22T01:59:52.284938+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:52.284938+0000","last_undegraded":"2024-04-22T01:59:52.284938+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"75'55","reported_seq":87,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.440956+0000","last_change":"2024-04-22T01:59:46.335573+0000","last_active":"2024-04-22T01:59:52.440956+0000","last_peered":"2024-04-22T01:59:52.440956+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.255441+0000","last_became_peered":"2024-04-22T01:59:30.255441+0000","last_unstale":"2024-04-22T01:59:52.440956+0000","last_undegraded":"2024-04-22T01:59:46.315293+0000","last_fullsized":"2024-04-22T01:59:30.161229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.307278+0000","last_change":"2024-04-22T01:59:45.189059+0000","last_active":"2024-04-22T01:59:53.307278+0000","last_peered":"2024-04-22T01:59:53.307278+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:45.189059+0000","last_became_peered":"2024-04-22T01:59:45.189059+0000","last_unstale":"2024-04-22T01:59:53.307278+0000","last_undegraded":"2024-04-22T01:59:53.307278+0000","last_fullsized":"2024-04-22T01:59:45.083825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"74'80","reported_seq":111,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.238498+0000","last_change":"2024-04-22T01:59:46.737480+0000","last_active":"2024-04-22T01:59:52.238498+0000","last_peered":"2024-04-22T01:59:52.238498+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.236750+0000","last_became_peered":"2024-04-22T01:59:30.236750+0000","last_unstale":"2024-04-22T01:59:52.238498+0000","last_undegraded":"2024-04-22T01:59:46.716626+0000","last_fullsized":"2024-04-22T01:59:30.165443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.254212+0000","last_change":"2024-04-22T01:59:45.223952+0000","last_active":"2024-04-22T01:59:52.254212+0000","last_peered":"2024-04-22T01:59:52.254212+0000","last_clean":"2024-04-22T01:59:52.254212+0000","last_became_active":"2024-04-22T01:59:45.217864+0000","last_became_peered":"2024-04-22T01:59:45.217864+0000","last_unstale":"2024-04-22T01:59:52.254212+0000","last_undegraded":"2024-04-22T01:59:52.254212+0000","last_fullsized":"2024-04-22T01:59:52.254212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.586623+0000","last_change":"2024-04-22T01:59:46.359994+0000","last_active":"2024-04-22T01:59:53.586623+0000","last_peered":"2024-04-22T01:59:53.586623+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.149551+0000","last_became_peered":"2024-04-22T01:59:45.149551+0000","last_unstale":"2024-04-22T01:59:53.586623+0000","last_undegraded":"2024-04-22T01:59:46.345979+0000","last_fullsized":"2024-04-22T01:59:45.075218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":196,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.295356+0000","last_change":"2024-04-22T01:59:45.178441+0000","last_active":"2024-04-22T01:59:52.295356+0000","last_peered":"2024-04-22T01:59:52.295356+0000","last_clean":"2024-04-22T01:59:52.295356+0000","last_became_active":"2024-04-22T01:59:45.171797+0000","last_became_peered":"2024-04-22T01:59:45.171797+0000","last_unstale":"2024-04-22T01:59:52.295356+0000","last_undegraded":"2024-04-22T01:59:52.295356+0000","last_fullsized":"2024-04-22T01:59:52.295356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"75'39","reported_seq":74,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.100195+0000","last_change":"2024-04-22T01:59:45.165903+0000","last_active":"2024-04-22T01:59:53.100195+0000","last_peered":"2024-04-22T01:59:53.100195+0000","last_clean":"2024-04-22T01:59:53.100195+0000","last_became_active":"2024-04-22T01:59:45.156195+0000","last_became_peered":"2024-04-22T01:59:45.156195+0000","last_unstale":"2024-04-22T01:59:53.100195+0000","last_undegraded":"2024-04-22T01:59:53.100195+0000","last_fullsized":"2024-04-22T01:59:53.100195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.241805+0000","last_change":"2024-04-22T01:59:45.241888+0000","last_active":"2024-04-22T01:59:52.241805+0000","last_peered":"2024-04-22T01:59:52.241805+0000","last_clean":"2024-04-22T01:59:52.241805+0000","last_became_active":"2024-04-22T01:59:45.236660+0000","last_became_peered":"2024-04-22T01:59:45.236660+0000","last_unstale":"2024-04-22T01:59:52.241805+0000","last_undegraded":"2024-04-22T01:59:52.241805+0000","last_fullsized":"2024-04-22T01:59:52.241805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"76'103","reported_seq":126,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.667754+0000","last_change":"2024-04-22T01:59:46.294155+0000","last_active":"2024-04-22T01:59:53.667754+0000","last_peered":"2024-04-22T01:59:53.667754+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.194515+0000","last_became_peered":"2024-04-22T01:59:45.194515+0000","last_unstale":"2024-04-22T01:59:53.667754+0000","last_undegraded":"2024-04-22T01:59:46.287653+0000","last_fullsized":"2024-04-22T01:59:45.080203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.294010+0000","last_change":"2024-04-22T01:59:45.203729+0000","last_active":"2024-04-22T01:59:53.294010+0000","last_peered":"2024-04-22T01:59:53.294010+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:45.203729+0000","last_became_peered":"2024-04-22T01:59:45.203729+0000","last_unstale":"2024-04-22T01:59:53.294010+0000","last_undegraded":"2024-04-22T01:59:53.294010+0000","last_fullsized":"2024-04-22T01:59:45.078142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":9641984,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":9416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1169,"ondisk_log_size":1169,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34740,"kb_used_data":3756,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94337100,"statfs":{"total":96636764160,"available":96601190400,"internally_reserved":0,"allocated":3846144,"data_stored":7147648,"data_compressed":355268,"data_compressed_allocated":3198976,"data_compressed_original":6889472,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39484,"kb_used_data":4020,"kb_used_omap":0,"kb_used_meta":35456,"kb_avail":94332356,"statfs":{"total":96636764160,"available":96596332544,"internally_reserved":0,"allocated":4116480,"data_stored":7701459,"data_compressed":383962,"data_compressed_allocated":3477504,"data_compressed_original":7446528,"omap_allocated":0,"internal_metadata":36306944},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":72,"seq":309237645321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31836,"kb_used_data":4372,"kb_used_omap":0,"kb_used_meta":27456,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":4476928,"data_stored":8409216,"data_compressed":420372,"data_compressed_allocated":3829760,"data_compressed_original":8151040,"omap_allocated":0,"internal_metadata":28114944},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:56.059 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-22T01:59:56.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T01:59:56.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:59:56.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:56.171+0000 7fce98d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:56.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:56.179+0000 7fce98d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:56.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 1667 1651 0.515888 1.21094 0.120488 0.241793 2024-04-22T01:59:56.425 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-04-22T01:59:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-04-22T01:59:56.442 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-04-22T01:59:56.243957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10232352,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1314,"num_write_kb":11310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1266,"ondisk_log_size":1266,"up":60,"acting":60,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":158744,"kb_used_data":25144,"kb_used_omap":0,"kb_used_meta":133568,"kb_avail":377328616,"statfs":{"total":386547056640,"available":386384502784,"internally_reserved":0,"allocated":25747456,"data_stored":49154949,"data_compressed":2487303,"data_compressed_allocated":23371776,"data_compressed_original":48218112,"omap_allocated":0,"internal_metadata":136773632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6463488,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":6312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044573"},"pg_stats":[{"pgid":"5.f","version":"74'56","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249692+0000","last_change":"2024-04-22T01:59:45.265202+0000","last_active":"2024-04-22T01:59:52.249692+0000","last_peered":"2024-04-22T01:59:52.249692+0000","last_clean":"2024-04-22T01:59:52.249692+0000","last_became_active":"2024-04-22T01:59:45.259408+0000","last_became_peered":"2024-04-22T01:59:45.259408+0000","last_unstale":"2024-04-22T01:59:52.249692+0000","last_undegraded":"2024-04-22T01:59:52.249692+0000","last_fullsized":"2024-04-22T01:59:52.249692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"76'59","reported_seq":87,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.797741+0000","last_change":"2024-04-22T01:59:45.221309+0000","last_active":"2024-04-22T01:59:53.797741+0000","last_peered":"2024-04-22T01:59:53.797741+0000","last_clean":"2024-04-22T01:59:53.797741+0000","last_became_active":"2024-04-22T01:59:45.213752+0000","last_became_peered":"2024-04-22T01:59:45.213752+0000","last_unstale":"2024-04-22T01:59:53.797741+0000","last_undegraded":"2024-04-22T01:59:53.797741+0000","last_fullsized":"2024-04-22T01:59:53.797741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60,"num_write_kb":480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"75'112","reported_seq":139,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.291542+0000","last_change":"2024-04-22T01:59:45.200056+0000","last_active":"2024-04-22T01:59:53.291542+0000","last_peered":"2024-04-22T01:59:53.291542+0000","last_clean":"2024-04-22T01:59:53.291542+0000","last_became_active":"2024-04-22T01:59:45.188317+0000","last_became_peered":"2024-04-22T01:59:45.188317+0000","last_unstale":"2024-04-22T01:59:53.291542+0000","last_undegraded":"2024-04-22T01:59:53.291542+0000","last_fullsized":"2024-04-22T01:59:53.291542+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":917504,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"75'92","reported_seq":124,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.176755+0000","last_change":"2024-04-22T01:59:31.285251+0000","last_active":"2024-04-22T01:59:53.176755+0000","last_peered":"2024-04-22T01:59:53.176755+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.253372+0000","last_became_peered":"2024-04-22T01:59:30.253372+0000","last_unstale":"2024-04-22T01:59:53.176755+0000","last_undegraded":"2024-04-22T01:59:31.273058+0000","last_fullsized":"2024-04-22T01:59:30.169001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":761856,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"75'136","reported_seq":158,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.295501+0000","last_change":"2024-04-22T01:59:46.301600+0000","last_active":"2024-04-22T01:59:53.295501+0000","last_peered":"2024-04-22T01:59:53.295501+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.153369+0000","last_became_peered":"2024-04-22T01:59:45.153369+0000","last_unstale":"2024-04-22T01:59:53.295501+0000","last_undegraded":"2024-04-22T01:59:46.288211+0000","last_fullsized":"2024-04-22T01:59:45.081008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":17}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"74'48","reported_seq":82,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.252166+0000","last_change":"2024-04-22T01:59:45.247767+0000","last_active":"2024-04-22T01:59:52.252166+0000","last_peered":"2024-04-22T01:59:52.252166+0000","last_clean":"2024-04-22T01:59:52.252166+0000","last_became_active":"2024-04-22T01:59:45.242401+0000","last_became_peered":"2024-04-22T01:59:45.242401+0000","last_unstale":"2024-04-22T01:59:52.252166+0000","last_undegraded":"2024-04-22T01:59:52.252166+0000","last_fullsized":"2024-04-22T01:59:52.252166+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.318994+0000","last_change":"2024-04-22T01:59:49.041718+0000","last_active":"2024-04-22T01:59:53.318994+0000","last_peered":"2024-04-22T01:59:53.318994+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.207850+0000","last_became_peered":"2024-04-22T01:59:45.207850+0000","last_unstale":"2024-04-22T01:59:53.318994+0000","last_undegraded":"2024-04-22T01:59:49.025876+0000","last_fullsized":"2024-04-22T01:59:45.092045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"75'40","reported_seq":62,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.274559+0000","last_change":"2024-04-22T01:59:47.695363+0000","last_active":"2024-04-22T01:59:53.274559+0000","last_peered":"2024-04-22T01:59:53.274559+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.155260+0000","last_became_peered":"2024-04-22T01:59:45.155260+0000","last_unstale":"2024-04-22T01:59:53.274559+0000","last_undegraded":"2024-04-22T01:59:47.678787+0000","last_fullsized":"2024-04-22T01:59:45.058562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"75'47","reported_seq":78,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.362923+0000","last_change":"2024-04-22T01:59:46.595867+0000","last_active":"2024-04-22T01:59:52.362923+0000","last_peered":"2024-04-22T01:59:52.362923+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.198798+0000","last_became_peered":"2024-04-22T01:59:45.198798+0000","last_unstale":"2024-04-22T01:59:52.362923+0000","last_undegraded":"2024-04-22T01:59:46.578328+0000","last_fullsized":"2024-04-22T01:59:45.066794+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":332,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.288619+0000","last_change":"2024-04-22T01:59:45.161116+0000","last_active":"2024-04-22T01:59:52.288619+0000","last_peered":"2024-04-22T01:59:52.288619+0000","last_clean":"2024-04-22T01:59:52.288619+0000","last_became_active":"2024-04-22T01:59:45.150739+0000","last_became_peered":"2024-04-22T01:59:45.150739+0000","last_unstale":"2024-04-22T01:59:52.288619+0000","last_undegraded":"2024-04-22T01:59:52.288619+0000","last_fullsized":"2024-04-22T01:59:52.288619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:24.245167+0000","last_clean_scrub_stamp":"2024-04-22T01:55:24.245167+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.278096+0000","last_change":"2024-04-22T01:59:45.165595+0000","last_active":"2024-04-22T01:59:53.278096+0000","last_peered":"2024-04-22T01:59:53.278096+0000","last_clean":"2024-04-22T01:57:37.080225+0000","last_became_active":"2024-04-22T01:59:45.165595+0000","last_became_peered":"2024-04-22T01:59:45.165595+0000","last_unstale":"2024-04-22T01:59:53.278096+0000","last_undegraded":"2024-04-22T01:59:53.278096+0000","last_fullsized":"2024-04-22T01:59:45.065478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"74'80","reported_seq":102,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.283368+0000","last_change":"2024-04-22T01:59:47.025432+0000","last_active":"2024-04-22T01:59:53.283368+0000","last_peered":"2024-04-22T01:59:53.283368+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.183869+0000","last_became_peered":"2024-04-22T01:59:45.183869+0000","last_unstale":"2024-04-22T01:59:53.283368+0000","last_undegraded":"2024-04-22T01:59:47.005374+0000","last_fullsized":"2024-04-22T01:59:45.076865+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.297858+0000","last_change":"2024-04-22T01:59:45.198972+0000","last_active":"2024-04-22T01:59:53.297858+0000","last_peered":"2024-04-22T01:59:53.297858+0000","last_clean":"2024-04-22T01:59:00.163293+0000","last_became_active":"2024-04-22T01:59:45.198972+0000","last_became_peered":"2024-04-22T01:59:45.198972+0000","last_unstale":"2024-04-22T01:59:53.297858+0000","last_undegraded":"2024-04-22T01:59:53.297858+0000","last_fullsized":"2024-04-22T01:59:45.072137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"74'80","reported_seq":114,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.249333+0000","last_change":"2024-04-22T01:59:45.214443+0000","last_active":"2024-04-22T01:59:52.249333+0000","last_peered":"2024-04-22T01:59:52.249333+0000","last_clean":"2024-04-22T01:59:52.249333+0000","last_became_active":"2024-04-22T01:59:45.205299+0000","last_became_peered":"2024-04-22T01:59:45.205299+0000","last_unstale":"2024-04-22T01:59:52.249333+0000","last_undegraded":"2024-04-22T01:59:52.249333+0000","last_fullsized":"2024-04-22T01:59:52.249333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":125,"reported_epoch":75,"state":"active+undersized","last_fresh":"2024-04-22T01:59:52.284938+0000","last_change":"2024-04-22T01:59:18.181471+0000","last_active":"2024-04-22T01:59:52.284938+0000","last_peered":"2024-04-22T01:59:52.284938+0000","last_clean":"2024-04-22T01:57:37.077569+0000","last_became_active":"2024-04-22T01:59:18.181471+0000","last_became_peered":"2024-04-22T01:59:18.181471+0000","last_unstale":"2024-04-22T01:59:52.284938+0000","last_undegraded":"2024-04-22T01:59:52.284938+0000","last_fullsized":"2024-04-22T01:59:18.077960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"75'55","reported_seq":87,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.440956+0000","last_change":"2024-04-22T01:59:46.335573+0000","last_active":"2024-04-22T01:59:52.440956+0000","last_peered":"2024-04-22T01:59:52.440956+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.255441+0000","last_became_peered":"2024-04-22T01:59:30.255441+0000","last_unstale":"2024-04-22T01:59:52.440956+0000","last_undegraded":"2024-04-22T01:59:46.315293+0000","last_fullsized":"2024-04-22T01:59:30.161229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":146,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.307278+0000","last_change":"2024-04-22T01:59:45.189059+0000","last_active":"2024-04-22T01:59:53.307278+0000","last_peered":"2024-04-22T01:59:53.307278+0000","last_clean":"2024-04-22T01:59:00.158431+0000","last_became_active":"2024-04-22T01:59:45.189059+0000","last_became_peered":"2024-04-22T01:59:45.189059+0000","last_unstale":"2024-04-22T01:59:53.307278+0000","last_undegraded":"2024-04-22T01:59:53.307278+0000","last_fullsized":"2024-04-22T01:59:45.083825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"74'80","reported_seq":111,"reported_epoch":75,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:52.238498+0000","last_change":"2024-04-22T01:59:46.737480+0000","last_active":"2024-04-22T01:59:52.238498+0000","last_peered":"2024-04-22T01:59:52.238498+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:30.236750+0000","last_became_peered":"2024-04-22T01:59:30.236750+0000","last_unstale":"2024-04-22T01:59:52.238498+0000","last_undegraded":"2024-04-22T01:59:46.716626+0000","last_fullsized":"2024-04-22T01:59:30.165443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.254212+0000","last_change":"2024-04-22T01:59:45.223952+0000","last_active":"2024-04-22T01:59:52.254212+0000","last_peered":"2024-04-22T01:59:52.254212+0000","last_clean":"2024-04-22T01:59:52.254212+0000","last_became_active":"2024-04-22T01:59:45.217864+0000","last_became_peered":"2024-04-22T01:59:45.217864+0000","last_unstale":"2024-04-22T01:59:52.254212+0000","last_undegraded":"2024-04-22T01:59:52.254212+0000","last_fullsized":"2024-04-22T01:59:52.254212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"76'71","reported_seq":94,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.586623+0000","last_change":"2024-04-22T01:59:46.359994+0000","last_active":"2024-04-22T01:59:53.586623+0000","last_peered":"2024-04-22T01:59:53.586623+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.149551+0000","last_became_peered":"2024-04-22T01:59:45.149551+0000","last_unstale":"2024-04-22T01:59:53.586623+0000","last_undegraded":"2024-04-22T01:59:46.345979+0000","last_fullsized":"2024-04-22T01:59:45.075218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72,"num_write_kb":576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":196,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.295356+0000","last_change":"2024-04-22T01:59:45.178441+0000","last_active":"2024-04-22T01:59:52.295356+0000","last_peered":"2024-04-22T01:59:52.295356+0000","last_clean":"2024-04-22T01:59:52.295356+0000","last_became_active":"2024-04-22T01:59:45.171797+0000","last_became_peered":"2024-04-22T01:59:45.171797+0000","last_unstale":"2024-04-22T01:59:52.295356+0000","last_undegraded":"2024-04-22T01:59:52.295356+0000","last_fullsized":"2024-04-22T01:59:52.295356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"75'39","reported_seq":74,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:53.100195+0000","last_change":"2024-04-22T01:59:45.165903+0000","last_active":"2024-04-22T01:59:53.100195+0000","last_peered":"2024-04-22T01:59:53.100195+0000","last_clean":"2024-04-22T01:59:53.100195+0000","last_became_active":"2024-04-22T01:59:45.156195+0000","last_became_peered":"2024-04-22T01:59:45.156195+0000","last_unstale":"2024-04-22T01:59:53.100195+0000","last_undegraded":"2024-04-22T01:59:53.100195+0000","last_fullsized":"2024-04-22T01:59:53.100195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":137,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-22T01:59:52.241805+0000","last_change":"2024-04-22T01:59:45.241888+0000","last_active":"2024-04-22T01:59:52.241805+0000","last_peered":"2024-04-22T01:59:52.241805+0000","last_clean":"2024-04-22T01:59:52.241805+0000","last_became_active":"2024-04-22T01:59:45.236660+0000","last_became_peered":"2024-04-22T01:59:45.236660+0000","last_unstale":"2024-04-22T01:59:52.241805+0000","last_undegraded":"2024-04-22T01:59:52.241805+0000","last_fullsized":"2024-04-22T01:59:52.241805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"76'103","reported_seq":126,"reported_epoch":76,"state":"active+undersized+degraded","last_fresh":"2024-04-22T01:59:53.667754+0000","last_change":"2024-04-22T01:59:46.294155+0000","last_active":"2024-04-22T01:59:53.667754+0000","last_peered":"2024-04-22T01:59:53.667754+0000","last_clean":"2024-04-22T01:59:29.137018+0000","last_became_active":"2024-04-22T01:59:45.194515+0000","last_became_peered":"2024-04-22T01:59:45.194515+0000","last_unstale":"2024-04-22T01:59:53.667754+0000","last_undegraded":"2024-04-22T01:59:46.287653+0000","last_fullsized":"2024-04-22T01:59:45.080203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:59:29.137018+0000","last_clean_scrub_stamp":"2024-04-22T01:59:29.137018+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":76,"state":"active+undersized","last_fresh":"2024-04-22T01:59:53.294010+0000","last_change":"2024-04-22T01:59:45.203729+0000","last_active":"2024-04-22T01:59:53.294010+0000","last_peered":"2024-04-22T01:59:53.294010+0000","last_clean":"2024-04-22T01:57:37.074750+0000","last_became_active":"2024-04-22T01:59:45.203729+0000","last_became_peered":"2024-04-22T01:59:45.203729+0000","last_unstale":"2024-04-22T01:59:53.294010+0000","last_undegraded":"2024-04-22T01:59:53.294010+0000","last_fullsized":"2024-04-22T01:59:45.078142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:55:27.252369+0000","last_clean_scrub_stamp":"2024-04-22T01:55:27.252369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":9641984,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":9416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1169,"ondisk_log_size":1169,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34740,"kb_used_data":3756,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94337100,"statfs":{"total":96636764160,"available":96601190400,"internally_reserved":0,"allocated":3846144,"data_stored":7147648,"data_compressed":355268,"data_compressed_allocated":3198976,"data_compressed_original":6889472,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":52,"seq":223338299457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39484,"kb_used_data":4020,"kb_used_omap":0,"kb_used_meta":35456,"kb_avail":94332356,"statfs":{"total":96636764160,"available":96596332544,"internally_reserved":0,"allocated":4116480,"data_stored":7701459,"data_compressed":383962,"data_compressed_allocated":3477504,"data_compressed_original":7446528,"omap_allocated":0,"internal_metadata":36306944},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52684,"kb_used_data":12996,"kb_used_omap":0,"kb_used_meta":39680,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":13307904,"data_stored":25896626,"data_compressed":1327701,"data_compressed_allocated":12865536,"data_compressed_original":25731072,"omap_allocated":0,"internal_metadata":40632320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":72,"seq":309237645321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31836,"kb_used_data":4372,"kb_used_omap":0,"kb_used_meta":27456,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":4476928,"data_stored":8409216,"data_compressed":420372,"data_compressed_allocated":3829760,"data_compressed_original":8151040,"omap_allocated":0,"internal_metadata":28114944},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:59:56.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T01:59:56.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:56.963+0000 7ff7c0f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:56.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:56.970+0000 7ff7c0f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:57.010 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:57.077 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T01:59:57.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 1827 1811 0.544119 1.25 0.145529 0.229135 2024-04-22T01:59:58.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 1994 1978 0.572284 1.30469 0.156686 0.217649 2024-04-22T01:59:59.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T01:59:59.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:59.205+0000 7f57d066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T01:59:59.211+0000 7f57d066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:59:59.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 2156 2140 0.597042 1.26562 0.124923 0.208934 2024-04-22T01:59:59.237 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T01:59:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T01:59:59.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:00:00.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~66s 2024-04-22T02:00:00.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 2308 2292 0.617399 1.1875 0.0435311 0.20203 2024-04-22T02:00:01.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 2468 2452 0.638482 1.25 0.0696687 0.195337 2024-04-22T02:00:01.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:00:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:01.440+0000 7ff793b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:01.444 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [3, 1, 2, 0] 2024-04-22T02:00:01.444 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T02:00:01.444 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [3, 1, 2, 0] 2024-04-22T02:00:01.445 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T02:00:01.445 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:00:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:01.446+0000 7ff793b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:01.470 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T02:00:01.530 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T02:00:02.232 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 2502 2486 0.626452 0.265625 0.0991833 0.194117 2024-04-22T02:00:03.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 2502 2486 0.606875 0 - 0.194117 2024-04-22T02:00:04.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 2502 2486 0.588485 0 - 0.194117 2024-04-22T02:00:05.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 2502 2486 0.571175 0 - 0.194117 2024-04-22T02:00:05.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~71s 2024-04-22T02:00:06.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 2502 2486 0.554856 0 - 0.194117 2024-04-22T02:00:06.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [3, 1, 0] 2024-04-22T02:00:06.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T02:00:06.531 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [3, 1, 0] 2024-04-22T02:00:06.531 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T02:00:06.531 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:00:06.606 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T02:00:07.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 2502 2486 0.539442 0 - 0.194117 2024-04-22T02:00:08.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 2502 2486 0.524862 0 - 0.194117 2024-04-22T02:00:09.233 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 2502 2486 0.511049 0 - 0.194117 2024-04-22T02:00:10.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 2502 2486 0.497945 0 - 0.194117 2024-04-22T02:00:10.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2024-04-22T02:00:11.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:00:11.235917+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:00:11.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:00:11.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 2502 2486 0.485495 0 - 0.194117 2024-04-22T02:00:11.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0] 2024-04-22T02:00:11.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T02:00:11.608 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T02:00:11.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T02:00:11.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.725+0000 7febf275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:11.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.734+0000 7febf275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:config help 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:config show 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:deep_scrub 2024-04-22T02:00:11.749 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_ops 2024-04-22T02:00:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:dump_historic_slow_ops 2024-04-22T02:00:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:dump_metrics [] 2024-04-22T02:00:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:dump_ops_in_flight 2024-04-22T02:00:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:dump_pgstate_history 2024-04-22T02:00:11.750 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-04-22T02:00:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:00:11.753 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:00:11.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T02:00:11.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.856+0000 7f3edc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:11.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.864+0000 7f3edc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:11.868 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T02:00:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-22T02:00:11.871 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:00:11.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T02:00:11.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.973+0000 7fd4f139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:00:11.982+0000 7fd4f139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:00:12.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 2502 2486 0.473653 0 - 0.194117 2024-04-22T02:00:13.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 2502 2486 0.462375 0 - 0.194117 2024-04-22T02:00:14.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 2502 2486 0.451622 0 - 0.194117 2024-04-22T02:00:15.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 2502 2486 0.441357 0 - 0.194117 2024-04-22T02:00:16.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~82s 2024-04-22T02:00:16.234 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 2502 2486 0.431549 0 - 0.194117 2024-04-22T02:00:17.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 2502 2486 0.422168 0 - 0.194117 2024-04-22T02:00:18.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 2502 2486 0.413185 0 - 0.194117 2024-04-22T02:00:19.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 2502 2486 0.404577 0 - 0.194117 2024-04-22T02:00:20.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 2502 2486 0.396319 0 - 0.194117 2024-04-22T02:00:21.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 2502 2486 0.388393 0 - 0.194117 2024-04-22T02:00:21.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~87s 2024-04-22T02:00:22.235 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 2502 2486 0.380777 0 - 0.194117 2024-04-22T02:00:23.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 2502 2486 0.373454 0 - 0.194117 2024-04-22T02:00:24.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 2502 2486 0.366408 0 - 0.194117 2024-04-22T02:00:25.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 2502 2486 0.359622 0 - 0.194117 2024-04-22T02:00:26.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 2502 2486 0.353084 0 - 0.194117 2024-04-22T02:00:26.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~93s 2024-04-22T02:00:27.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 2502 2486 0.346779 0 - 0.194117 2024-04-22T02:00:28.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 2502 2486 0.340694 0 - 0.194117 2024-04-22T02:00:29.236 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 2502 2486 0.33482 0 - 0.194117 2024-04-22T02:00:30.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 2502 2486 0.329145 0 - 0.194117 2024-04-22T02:00:31.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:00:31.238794+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:00:31.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:00:31.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 2502 2486 0.323659 0 - 0.194117 2024-04-22T02:00:32.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~98s 2024-04-22T02:00:32.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 2502 2486 0.318353 0 - 0.194117 2024-04-22T02:00:33.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 2502 2486 0.313219 0 - 0.194117 2024-04-22T02:00:34.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 2502 2486 0.308247 0 - 0.194117 2024-04-22T02:00:35.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 2502 2486 0.30343 0 - 0.194117 2024-04-22T02:00:36.237 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 2502 2486 0.298762 0 - 0.194117 2024-04-22T02:00:37.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 2502 2486 0.294235 0 - 0.194117 2024-04-22T02:00:37.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~103s 2024-04-22T02:00:38.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 2502 2486 0.289843 0 - 0.194117 2024-04-22T02:00:39.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 2502 2486 0.285581 0 - 0.194117 2024-04-22T02:00:40.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 2502 2486 0.281442 0 - 0.194117 2024-04-22T02:00:41.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 2502 2486 0.277421 0 - 0.194117 2024-04-22T02:00:42.238 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 2502 2486 0.273514 0 - 0.194117 2024-04-22T02:00:42.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~109s 2024-04-22T02:00:43.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 2502 2486 0.269715 0 - 0.194117 2024-04-22T02:00:44.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 2502 2486 0.26602 0 - 0.194117 2024-04-22T02:00:45.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 2502 2486 0.262425 0 - 0.194117 2024-04-22T02:00:46.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 2502 2486 0.258926 0 - 0.194117 2024-04-22T02:00:47.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 2502 2486 0.255519 0 - 0.194117 2024-04-22T02:00:47.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~114s 2024-04-22T02:00:48.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 2502 2486 0.252201 0 - 0.194117 2024-04-22T02:00:49.239 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 2502 2486 0.248967 0 - 0.194117 2024-04-22T02:00:50.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 2502 2486 0.245816 0 - 0.194117 2024-04-22T02:00:51.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:00:51.241733+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:00:51.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:00:51.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 2502 2486 0.242743 0 - 0.194117 2024-04-22T02:00:52.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 2502 2486 0.239746 0 - 0.194117 2024-04-22T02:00:53.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 2502 2486 0.236822 0 - 0.194117 2024-04-22T02:00:53.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~119s 2024-04-22T02:00:54.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 2502 2486 0.233969 0 - 0.194117 2024-04-22T02:00:55.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 2502 2486 0.231183 0 - 0.194117 2024-04-22T02:00:56.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 2502 2486 0.228463 0 - 0.194117 2024-04-22T02:00:57.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 2502 2486 0.225807 0 - 0.194117 2024-04-22T02:00:58.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 2502 2486 0.223211 0 - 0.194117 2024-04-22T02:00:58.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~125s 2024-04-22T02:00:59.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 2502 2486 0.220675 0 - 0.194117 2024-04-22T02:01:00.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 2502 2486 0.218195 0 - 0.194117 2024-04-22T02:01:01.041 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:01.042+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:01.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 16 2502 2486 0.215771 0 - 0.194117 2024-04-22T02:01:02.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 91 16 2502 2486 0.2134 0 - 0.194117 2024-04-22T02:01:03.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 92 16 2502 2486 0.21108 0 - 0.194117 2024-04-22T02:01:03.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-04-22T02:01:04.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 93 16 2502 2486 0.20881 0 - 0.194117 2024-04-22T02:01:05.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 94 16 2502 2486 0.206589 0 - 0.194117 2024-04-22T02:01:06.042 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:06.043+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:06.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 95 16 2502 2486 0.204414 0 - 0.194117 2024-04-22T02:01:07.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 96 16 2502 2486 0.202285 0 - 0.194117 2024-04-22T02:01:08.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 97 16 2502 2486 0.200199 0 - 0.194117 2024-04-22T02:01:09.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-04-22T02:01:09.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 98 16 2502 2486 0.198156 0 - 0.194117 2024-04-22T02:01:10.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 99 16 2502 2486 0.196155 0 - 0.194117 2024-04-22T02:01:11.043 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:11.044+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:11.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:01:11.244788+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:01:11.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:01:11.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 100 16 2502 2486 0.194193 0 - 0.194117 2024-04-22T02:01:12.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 101 16 2502 2486 0.19227 0 - 0.194117 2024-04-22T02:01:13.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 102 16 2502 2486 0.190385 0 - 0.194117 2024-04-22T02:01:14.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 103 16 2502 2486 0.188537 0 - 0.194117 2024-04-22T02:01:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~141s 2024-04-22T02:01:15.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 104 16 2502 2486 0.186724 0 - 0.194117 2024-04-22T02:01:16.044 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:16.045+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:16.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 105 16 2502 2486 0.184946 0 - 0.194117 2024-04-22T02:01:17.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 106 16 2502 2486 0.183201 0 - 0.194117 2024-04-22T02:01:18.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 107 16 2502 2486 0.181489 0 - 0.194117 2024-04-22T02:01:19.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 108 16 2502 2486 0.179808 0 - 0.194117 2024-04-22T02:01:19.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-04-22T02:01:20.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 109 16 2502 2486 0.178159 0 - 0.194117 2024-04-22T02:01:21.045 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:21.046+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:21.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 110 16 2502 2486 0.176539 0 - 0.194117 2024-04-22T02:01:22.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 111 16 2502 2486 0.174949 0 - 0.194117 2024-04-22T02:01:23.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 112 16 2502 2486 0.173387 0 - 0.194117 2024-04-22T02:01:24.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 113 16 2502 2486 0.171852 0 - 0.194117 2024-04-22T02:01:25.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-04-22T02:01:25.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 114 16 2502 2486 0.170345 0 - 0.194117 2024-04-22T02:01:26.046 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:26.047+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:26.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 115 16 2502 2486 0.168863 0 - 0.194117 2024-04-22T02:01:27.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 116 16 2502 2486 0.167408 0 - 0.194117 2024-04-22T02:01:28.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 117 16 2502 2486 0.165977 0 - 0.194117 2024-04-22T02:01:29.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 118 16 2502 2486 0.16457 0 - 0.194117 2024-04-22T02:01:30.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 119 16 2502 2486 0.163187 0 - 0.194117 2024-04-22T02:01:30.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~156s 2024-04-22T02:01:31.047 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:31.047+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:31.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:01:31.247723+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:01:31.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:01:31.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 120 16 2502 2486 0.161827 0 - 0.194117 2024-04-22T02:01:32.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 121 16 2502 2486 0.16049 0 - 0.194117 2024-04-22T02:01:33.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 122 16 2502 2486 0.159174 0 - 0.194117 2024-04-22T02:01:34.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 123 16 2502 2486 0.15788 0 - 0.194117 2024-04-22T02:01:35.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 124 16 2502 2486 0.156607 0 - 0.194117 2024-04-22T02:01:35.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~162s 2024-04-22T02:01:36.048 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:36.048+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:36.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 125 16 2502 2486 0.155354 0 - 0.194117 2024-04-22T02:01:37.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 126 16 2502 2486 0.154121 0 - 0.194117 2024-04-22T02:01:38.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 127 16 2502 2486 0.152907 0 - 0.194117 2024-04-22T02:01:39.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 128 16 2502 2486 0.151713 0 - 0.194117 2024-04-22T02:01:40.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 129 16 2502 2486 0.150537 0 - 0.194117 2024-04-22T02:01:40.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~167s 2024-04-22T02:01:41.048 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:41.049+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:41.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 130 16 2502 2486 0.149379 0 - 0.194117 2024-04-22T02:01:42.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 131 16 2502 2486 0.148238 0 - 0.194117 2024-04-22T02:01:43.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 132 16 2502 2486 0.147115 0 - 0.194117 2024-04-22T02:01:44.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 133 16 2502 2486 0.146009 0 - 0.194117 2024-04-22T02:01:45.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 134 16 2502 2486 0.14492 0 - 0.194117 2024-04-22T02:01:46.049 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:46.050+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:46.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 135 16 2502 2486 0.143846 0 - 0.194117 2024-04-22T02:01:46.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~172s 2024-04-22T02:01:47.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 136 16 2502 2486 0.142789 0 - 0.194117 2024-04-22T02:01:48.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 137 16 2502 2486 0.141746 0 - 0.194117 2024-04-22T02:01:49.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 138 16 2502 2486 0.140719 0 - 0.194117 2024-04-22T02:01:50.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 139 16 2502 2486 0.139707 0 - 0.194117 2024-04-22T02:01:51.050 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:51.051+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:51.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:01:51.250716+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:01:51.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:01:51.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 140 16 2502 2486 0.138709 0 - 0.194117 2024-04-22T02:01:51.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~178s 2024-04-22T02:01:52.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 141 16 2502 2486 0.137725 0 - 0.194117 2024-04-22T02:01:53.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 142 16 2502 2486 0.136755 0 - 0.194117 2024-04-22T02:01:54.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 143 16 2502 2486 0.135799 0 - 0.194117 2024-04-22T02:01:55.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 144 16 2502 2486 0.134856 0 - 0.194117 2024-04-22T02:01:56.051 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:01:56.052+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:01:56.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 145 16 2502 2486 0.133926 0 - 0.194117 2024-04-22T02:01:56.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~183s 2024-04-22T02:01:57.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 146 16 2502 2486 0.133008 0 - 0.194117 2024-04-22T02:01:58.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 147 16 2502 2486 0.132103 0 - 0.194117 2024-04-22T02:01:59.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 148 16 2502 2486 0.131211 0 - 0.194117 2024-04-22T02:02:00.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 149 16 2502 2486 0.13033 0 - 0.194117 2024-04-22T02:02:01.052 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:01.053+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:01.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 150 16 2502 2486 0.129461 0 - 0.194117 2024-04-22T02:02:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:02:01.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:02:02.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~188s 2024-04-22T02:02:02.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 151 16 2502 2486 0.128604 0 - 0.194117 2024-04-22T02:02:03.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 152 16 2502 2486 0.127758 0 - 0.194117 2024-04-22T02:02:03.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:02:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:02:03.478+0000 7ffbd542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:02:03.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:02:03.483+0000 7ffbd542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:02:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-22T02:02:04.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 153 16 2502 2486 0.126923 0 - 0.194117 2024-04-22T02:02:05.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 154 16 2502 2486 0.126099 0 - 0.194117 2024-04-22T02:02:06.053 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:06.054+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:06.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 155 16 2502 2486 0.125285 0 - 0.194117 2024-04-22T02:02:07.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 156 16 2502 2486 0.124482 0 - 0.194117 2024-04-22T02:02:07.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~194s 2024-04-22T02:02:08.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 157 16 2502 2486 0.123689 0 - 0.194117 2024-04-22T02:02:09.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 158 16 2502 2486 0.122906 0 - 0.194117 2024-04-22T02:02:10.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 159 16 2502 2486 0.122133 0 - 0.194117 2024-04-22T02:02:11.054 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:11.054+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:11.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:02:11.253700+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:02:11.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:02:11.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 160 16 2502 2486 0.12137 0 - 0.194117 2024-04-22T02:02:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:02:11.900 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:02:11.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T02:02:12.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:02:12.015+0000 7fb75882d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:02:12.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:02:12.024+0000 7fb75882d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:02:12.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 161 16 2502 2486 0.120616 0 - 0.194117 2024-04-22T02:02:12.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~199s 2024-04-22T02:02:13.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 162 16 2502 2486 0.119872 0 - 0.194117 2024-04-22T02:02:14.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 163 16 2502 2486 0.119136 0 - 0.194117 2024-04-22T02:02:15.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 164 16 2502 2486 0.11841 0 - 0.194117 2024-04-22T02:02:16.054 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:16.055+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:16.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 165 16 2502 2486 0.117692 0 - 0.194117 2024-04-22T02:02:17.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 166 16 2502 2486 0.116983 0 - 0.194117 2024-04-22T02:02:18.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~204s 2024-04-22T02:02:18.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 167 16 2502 2486 0.116283 0 - 0.194117 2024-04-22T02:02:19.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 168 16 2502 2486 0.11559 0 - 0.194117 2024-04-22T02:02:20.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 169 16 2502 2486 0.114907 0 - 0.194117 2024-04-22T02:02:21.055 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:21.056+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:21.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 170 16 2502 2486 0.114231 0 - 0.194117 2024-04-22T02:02:22.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 171 16 2502 2486 0.113563 0 - 0.194117 2024-04-22T02:02:23.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 172 16 2502 2486 0.112902 0 - 0.194117 2024-04-22T02:02:23.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~209s 2024-04-22T02:02:24.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 173 16 2502 2486 0.11225 0 - 0.194117 2024-04-22T02:02:25.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 174 16 2502 2486 0.111605 0 - 0.194117 2024-04-22T02:02:26.056 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:26.057+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:26.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 175 16 2502 2486 0.110967 0 - 0.194117 2024-04-22T02:02:27.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 176 16 2502 2486 0.110336 0 - 0.194117 2024-04-22T02:02:28.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 177 16 2502 2486 0.109713 0 - 0.194117 2024-04-22T02:02:28.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2024-04-22T02:02:29.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 178 16 2502 2486 0.109097 0 - 0.194117 2024-04-22T02:02:30.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 179 16 2502 2486 0.108487 0 - 0.194117 2024-04-22T02:02:31.057 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:31.057+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:31.254 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:02:31.256535+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:02:31.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:02:31.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 180 16 2502 2486 0.107884 0 - 0.194117 2024-04-22T02:02:32.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 181 16 2502 2486 0.107288 0 - 0.194117 2024-04-22T02:02:33.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 182 16 2502 2486 0.106699 0 - 0.194117 2024-04-22T02:02:34.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2024-04-22T02:02:34.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 183 16 2502 2486 0.106116 0 - 0.194117 2024-04-22T02:02:35.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 184 16 2502 2486 0.105539 0 - 0.194117 2024-04-22T02:02:36.058 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:36.058+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:36.255 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 185 16 2502 2486 0.104969 0 - 0.194117 2024-04-22T02:02:37.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 186 16 2502 2486 0.104404 0 - 0.194117 2024-04-22T02:02:38.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 187 16 2502 2486 0.103846 0 - 0.194117 2024-04-22T02:02:39.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 188 16 2502 2486 0.103294 0 - 0.194117 2024-04-22T02:02:39.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~225s 2024-04-22T02:02:40.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 189 16 2502 2486 0.102747 0 - 0.194117 2024-04-22T02:02:41.058 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:41.059+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:41.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 190 16 2502 2486 0.102206 0 - 0.194117 2024-04-22T02:02:42.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 191 16 2502 2486 0.101671 0 - 0.194117 2024-04-22T02:02:43.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 192 16 2502 2486 0.101142 0 - 0.194117 2024-04-22T02:02:44.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 193 16 2502 2486 0.100618 0 - 0.194117 2024-04-22T02:02:44.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2024-04-22T02:02:45.256 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 194 16 2502 2486 0.100099 0 - 0.194117 2024-04-22T02:02:46.060 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:46.060+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:46.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 195 16 2502 2486 0.0995855 0 - 0.194117 2024-04-22T02:02:47.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 196 16 2502 2486 0.0990774 0 - 0.194117 2024-04-22T02:02:48.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 197 16 2502 2486 0.0985745 0 - 0.194117 2024-04-22T02:02:49.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 198 16 2502 2486 0.0980767 0 - 0.194117 2024-04-22T02:02:49.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~236s 2024-04-22T02:02:50.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 199 16 2502 2486 0.0975838 0 - 0.194117 2024-04-22T02:02:51.061 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:51.061+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:51.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:02:51.259378+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:02:51.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:02:51.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 200 16 2502 2486 0.0970959 0 - 0.194117 2024-04-22T02:02:52.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 201 16 2502 2486 0.0966128 0 - 0.194117 2024-04-22T02:02:53.257 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 202 16 2502 2486 0.0961346 0 - 0.194117 2024-04-22T02:02:54.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 203 16 2502 2486 0.095661 0 - 0.194117 2024-04-22T02:02:55.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~241s 2024-04-22T02:02:55.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 204 16 2502 2486 0.0951921 0 - 0.194117 2024-04-22T02:02:56.061 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:02:56.062+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:02:56.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 205 16 2502 2486 0.0947277 0 - 0.194117 2024-04-22T02:02:57.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 206 16 2502 2486 0.0942679 0 - 0.194117 2024-04-22T02:02:58.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 207 16 2502 2486 0.0938125 0 - 0.194117 2024-04-22T02:02:59.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 208 16 2502 2486 0.0933614 0 - 0.194117 2024-04-22T02:03:00.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 209 16 2502 2486 0.0929147 0 - 0.194117 2024-04-22T02:03:00.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~247s 2024-04-22T02:03:01.062 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:01.063+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:01.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 210 16 2502 2486 0.0924723 0 - 0.194117 2024-04-22T02:03:02.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 211 16 2502 2486 0.092034 0 - 0.194117 2024-04-22T02:03:03.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 212 16 2502 2486 0.0915999 0 - 0.194117 2024-04-22T02:03:04.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 213 16 2502 2486 0.0911698 0 - 0.194117 2024-04-22T02:03:05.259 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 214 16 2502 2486 0.0907438 0 - 0.194117 2024-04-22T02:03:05.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~252s 2024-04-22T02:03:06.063 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:06.064+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:06.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 215 16 2502 2486 0.0903217 0 - 0.194117 2024-04-22T02:03:07.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 216 16 2502 2486 0.0899035 0 - 0.194117 2024-04-22T02:03:08.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 217 16 2502 2486 0.0894892 0 - 0.194117 2024-04-22T02:03:09.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 218 16 2502 2486 0.0890787 0 - 0.194117 2024-04-22T02:03:10.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 219 16 2502 2486 0.088672 0 - 0.194117 2024-04-22T02:03:11.064 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:11.065+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:11.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~257s 2024-04-22T02:03:11.260 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:03:11.262343+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:03:11.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:03:11.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 220 16 2502 2486 0.0882689 0 - 0.194117 2024-04-22T02:03:12.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 221 16 2502 2486 0.0878695 0 - 0.194117 2024-04-22T02:03:13.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 222 16 2502 2486 0.0874737 0 - 0.194117 2024-04-22T02:03:14.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 223 16 2502 2486 0.0870814 0 - 0.194117 2024-04-22T02:03:15.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 224 16 2502 2486 0.0866927 0 - 0.194117 2024-04-22T02:03:16.066 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:16.067+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:16.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 225 16 2502 2486 0.0863074 0 - 0.194117 2024-04-22T02:03:16.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~262s 2024-04-22T02:03:17.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 226 16 2502 2486 0.0859255 0 - 0.194117 2024-04-22T02:03:18.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 227 16 2502 2486 0.0855469 0 - 0.194117 2024-04-22T02:03:19.261 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 228 16 2502 2486 0.0851717 0 - 0.194117 2024-04-22T02:03:20.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 229 16 2502 2486 0.0847998 0 - 0.194117 2024-04-22T02:03:21.067 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:21.067+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:21.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 230 16 2502 2486 0.0844311 0 - 0.194117 2024-04-22T02:03:21.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~268s 2024-04-22T02:03:22.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 231 16 2502 2486 0.0840656 0 - 0.194117 2024-04-22T02:03:23.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 232 16 2502 2486 0.0837033 0 - 0.194117 2024-04-22T02:03:24.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 233 16 2502 2486 0.083344 0 - 0.194117 2024-04-22T02:03:25.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 234 16 2502 2486 0.0829879 0 - 0.194117 2024-04-22T02:03:26.068 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:26.067+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:26.262 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 235 16 2502 2486 0.0826347 0 - 0.194117 2024-04-22T02:03:27.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~273s 2024-04-22T02:03:27.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 236 16 2502 2486 0.0822846 0 - 0.194117 2024-04-22T02:03:28.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 237 16 2502 2486 0.0819374 0 - 0.194117 2024-04-22T02:03:29.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 238 16 2502 2486 0.0815931 0 - 0.194117 2024-04-22T02:03:30.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 239 16 2502 2486 0.0812517 0 - 0.194117 2024-04-22T02:03:31.068 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:31.068+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:31.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:03:31.265154+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:03:31.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:03:31.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 240 16 2502 2486 0.0809132 0 - 0.194117 2024-04-22T02:03:32.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 241 16 2502 2486 0.0805774 0 - 0.194117 2024-04-22T02:03:32.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~278s 2024-04-22T02:03:33.263 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 242 16 2502 2486 0.0802445 0 - 0.194117 2024-04-22T02:03:34.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 243 16 2502 2486 0.0799142 0 - 0.194117 2024-04-22T02:03:35.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 244 16 2502 2486 0.0795867 0 - 0.194117 2024-04-22T02:03:36.069 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:36.069+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:36.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 245 16 2502 2486 0.0792619 0 - 0.194117 2024-04-22T02:03:37.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 246 16 2502 2486 0.0789397 0 - 0.194117 2024-04-22T02:03:37.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~284s 2024-04-22T02:03:38.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 247 16 2502 2486 0.0786201 0 - 0.194117 2024-04-22T02:03:39.264 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 248 16 2502 2486 0.078303 0 - 0.194117 2024-04-22T02:03:40.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 249 16 2502 2486 0.0779886 0 - 0.194117 2024-04-22T02:03:41.070 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:41.071+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:41.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 250 16 2502 2486 0.0776766 0 - 0.194117 2024-04-22T02:03:42.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 251 16 2502 2486 0.0773671 0 - 0.194117 2024-04-22T02:03:42.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~289s 2024-04-22T02:03:43.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 252 16 2502 2486 0.0770601 0 - 0.194117 2024-04-22T02:03:44.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 253 16 2502 2486 0.0767556 0 - 0.194117 2024-04-22T02:03:45.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 254 16 2502 2486 0.0764534 0 - 0.194117 2024-04-22T02:03:46.071 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:46.072+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:46.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 255 16 2502 2486 0.0761535 0 - 0.194117 2024-04-22T02:03:47.265 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 256 16 2502 2486 0.0758561 0 - 0.194117 2024-04-22T02:03:48.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~294s 2024-04-22T02:03:48.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 257 16 2502 2486 0.0755609 0 - 0.194117 2024-04-22T02:03:49.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 258 16 2502 2486 0.075268 0 - 0.194117 2024-04-22T02:03:50.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 259 16 2502 2486 0.0749774 0 - 0.194117 2024-04-22T02:03:51.072 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:51.073+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:51.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:03:51.268092+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:03:51.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:03:51.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 260 16 2502 2486 0.074689 0 - 0.194117 2024-04-22T02:03:52.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 261 16 2502 2486 0.0744029 0 - 0.194117 2024-04-22T02:03:53.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 262 16 2502 2486 0.0741189 0 - 0.194117 2024-04-22T02:03:53.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~300s 2024-04-22T02:03:54.266 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 263 16 2502 2486 0.0738371 0 - 0.194117 2024-04-22T02:03:55.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 264 16 2502 2486 0.0735574 0 - 0.194117 2024-04-22T02:03:56.073 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:56.073+0000 7f6b1c1f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.2 v2:172.21.15.150:6803/253139254 for 29sec e78 v78) 2024-04-22T02:03:56.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 265 16 2502 2486 0.0732798 0 - 0.194117 2024-04-22T02:03:57.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 266 16 2502 2486 0.0730043 0 - 0.194117 2024-04-22T02:03:58.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 267 16 2502 2486 0.0727309 0 - 0.194117 2024-04-22T02:03:58.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~305s 2024-04-22T02:03:58.857 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-22T02:03:59.159 INFO:tasks.ceph.osd.0:Sent signal 15 2024-04-22T02:03:59.159 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-22T02:03:59.160 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-22T02:03:59.160 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:59.161+0000 7f6b1f1ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 27286) UID: 0 2024-04-22T02:03:59.160 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-04-22T02:03:59.161+0000 7f6b1f1ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T02:03:59.161 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T02:03:59.162+0000 7f3981df7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 26966) UID: 0 2024-04-22T02:03:59.161 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-04-22T02:03:59.162+0000 7f3981df7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T02:03:59.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 268 16 2502 2486 0.0724595 0 - 0.194117 2024-04-22T02:04:00.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 269 16 2502 2486 0.0721902 0 - 0.194117 2024-04-22T02:04:01.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 270 16 2502 2486 0.0719228 0 - 0.194117 2024-04-22T02:04:02.267 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 271 16 2502 2486 0.0716574 0 - 0.194117 2024-04-22T02:04:03.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 272 16 2502 2486 0.0713939 0 - 0.194117 2024-04-22T02:04:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:04:03.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:04:04.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 273 16 2502 2486 0.0711324 0 - 0.194117 2024-04-22T02:04:05.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 274 16 2502 2486 0.0708728 0 - 0.194117 2024-04-22T02:04:05.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:04:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:04:05.503+0000 7f43d379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:04:06.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 275 16 2502 2486 0.0706151 0 - 0.194117 2024-04-22T02:04:07.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 276 16 2502 2486 0.0703593 0 - 0.194117 2024-04-22T02:04:08.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 277 16 2502 2486 0.0701053 0 - 0.194117 2024-04-22T02:04:09.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 278 16 2502 2486 0.0698531 0 - 0.194117 2024-04-22T02:04:10.268 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 279 16 2502 2486 0.0696027 0 - 0.194117 2024-04-22T02:04:11.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:04:11.270775+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:04:11.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:04:11.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 280 16 2502 2486 0.0693541 0 - 0.194117 2024-04-22T02:04:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:04:11.935 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:04:12.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 281 16 2502 2486 0.0691073 0 - 0.194117 2024-04-22T02:04:13.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 282 16 2502 2486 0.0688623 0 - 0.194117 2024-04-22T02:04:14.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 283 16 2502 2486 0.068619 0 - 0.194117 2024-04-22T02:04:15.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 284 16 2502 2486 0.0683774 0 - 0.194117 2024-04-22T02:04:16.269 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 285 16 2502 2486 0.0681374 0 - 0.194117 2024-04-22T02:04:16.936 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0] 2024-04-22T02:04:16.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T02:04:16.937 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-04-22T02:04:16.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-04-22T02:04:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:04:17.052+0000 7f14d0a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:04:17.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 286 16 2502 2486 0.0678992 0 - 0.194117 2024-04-22T02:04:18.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 287 16 2502 2486 0.0676626 0 - 0.194117 2024-04-22T02:04:19.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 288 16 2502 2486 0.0674277 0 - 0.194117 2024-04-22T02:04:20.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 289 16 2502 2486 0.0671944 0 - 0.194117 2024-04-22T02:04:21.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 290 16 2502 2486 0.0669627 0 - 0.194117 2024-04-22T02:04:22.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 291 16 2502 2486 0.0667326 0 - 0.194117 2024-04-22T02:04:23.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 292 16 2502 2486 0.066504 0 - 0.194117 2024-04-22T02:04:24.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 293 16 2502 2486 0.066277 0 - 0.194117 2024-04-22T02:04:25.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 294 16 2502 2486 0.0660516 0 - 0.194117 2024-04-22T02:04:26.270 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 295 16 2502 2486 0.0658277 0 - 0.194117 2024-04-22T02:04:27.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 296 16 2502 2486 0.0656053 0 - 0.194117 2024-04-22T02:04:28.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 297 16 2502 2486 0.0653844 0 - 0.194117 2024-04-22T02:04:29.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 298 16 2502 2486 0.065165 0 - 0.194117 2024-04-22T02:04:30.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 299 16 2502 2486 0.0649471 0 - 0.194117 2024-04-22T02:04:31.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:04:31.273246+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:04:31.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:04:31.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 300 16 2502 2486 0.0647306 0 - 0.194117 2024-04-22T02:04:32.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 301 16 2502 2486 0.0645156 0 - 0.194117 2024-04-22T02:04:33.271 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 302 16 2502 2486 0.0643019 0 - 0.194117 2024-04-22T02:04:34.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 303 16 2502 2486 0.0640897 0 - 0.194117 2024-04-22T02:04:35.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 304 16 2502 2486 0.0638789 0 - 0.194117 2024-04-22T02:04:36.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 305 16 2502 2486 0.0636695 0 - 0.194117 2024-04-22T02:04:37.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 306 16 2502 2486 0.0634614 0 - 0.194117 2024-04-22T02:04:38.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 307 16 2502 2486 0.0632547 0 - 0.194117 2024-04-22T02:04:39.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 308 16 2502 2486 0.0630493 0 - 0.194117 2024-04-22T02:04:40.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 309 16 2502 2486 0.0628453 0 - 0.194117 2024-04-22T02:04:41.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 310 16 2502 2486 0.0626426 0 - 0.194117 2024-04-22T02:04:42.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 311 16 2502 2486 0.0624411 0 - 0.194117 2024-04-22T02:04:43.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 312 16 2502 2486 0.062241 0 - 0.194117 2024-04-22T02:04:44.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 313 16 2502 2486 0.0620422 0 - 0.194117 2024-04-22T02:04:45.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 314 16 2502 2486 0.0618446 0 - 0.194117 2024-04-22T02:04:46.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 315 16 2502 2486 0.0616482 0 - 0.194117 2024-04-22T02:04:47.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 316 16 2502 2486 0.0614531 0 - 0.194117 2024-04-22T02:04:48.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 317 16 2502 2486 0.0612593 0 - 0.194117 2024-04-22T02:04:49.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 318 16 2502 2486 0.0610667 0 - 0.194117 2024-04-22T02:04:50.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 319 16 2502 2486 0.0608752 0 - 0.194117 2024-04-22T02:04:51.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:04:51.275738+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:04:51.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:04:51.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 320 16 2502 2486 0.060685 0 - 0.194117 2024-04-22T02:04:52.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 321 16 2502 2486 0.0604959 0 - 0.194117 2024-04-22T02:04:53.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 322 16 2502 2486 0.0603081 0 - 0.194117 2024-04-22T02:04:54.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 323 16 2502 2486 0.0601214 0 - 0.194117 2024-04-22T02:04:55.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 324 16 2502 2486 0.0599358 0 - 0.194117 2024-04-22T02:04:56.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 325 16 2502 2486 0.0597514 0 - 0.194117 2024-04-22T02:04:57.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 326 16 2502 2486 0.0595681 0 - 0.194117 2024-04-22T02:04:58.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 327 16 2502 2486 0.0593859 0 - 0.194117 2024-04-22T02:04:59.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 328 16 2502 2486 0.0592049 0 - 0.194117 2024-04-22T02:05:00.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 329 16 2502 2486 0.0590249 0 - 0.194117 2024-04-22T02:05:01.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 330 16 2502 2486 0.0588461 0 - 0.194117 2024-04-22T02:05:02.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 331 16 2502 2486 0.0586683 0 - 0.194117 2024-04-22T02:05:03.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 332 16 2502 2486 0.0584916 0 - 0.194117 2024-04-22T02:05:04.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 333 16 2502 2486 0.0583159 0 - 0.194117 2024-04-22T02:05:05.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 334 16 2502 2486 0.0581413 0 - 0.194117 2024-04-22T02:05:06.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 335 16 2502 2486 0.0579678 0 - 0.194117 2024-04-22T02:05:07.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 336 16 2502 2486 0.0577953 0 - 0.194117 2024-04-22T02:05:08.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 337 16 2502 2486 0.0576238 0 - 0.194117 2024-04-22T02:05:09.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 338 16 2502 2486 0.0574533 0 - 0.194117 2024-04-22T02:05:10.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 339 16 2502 2486 0.0572838 0 - 0.194117 2024-04-22T02:05:11.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:05:11.278224+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:05:11.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:05:11.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 340 16 2502 2486 0.0571153 0 - 0.194117 2024-04-22T02:05:12.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 341 16 2502 2486 0.0569478 0 - 0.194117 2024-04-22T02:05:13.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 342 16 2502 2486 0.0567813 0 - 0.194117 2024-04-22T02:05:14.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 343 16 2502 2486 0.0566158 0 - 0.194117 2024-04-22T02:05:15.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 344 16 2502 2486 0.0564512 0 - 0.194117 2024-04-22T02:05:16.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 345 16 2502 2486 0.0562876 0 - 0.194117 2024-04-22T02:05:17.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 346 16 2502 2486 0.0561249 0 - 0.194117 2024-04-22T02:05:18.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 347 16 2502 2486 0.0559632 0 - 0.194117 2024-04-22T02:05:19.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 348 16 2502 2486 0.0558023 0 - 0.194117 2024-04-22T02:05:20.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 349 16 2502 2486 0.0556425 0 - 0.194117 2024-04-22T02:05:21.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 350 16 2502 2486 0.0554835 0 - 0.194117 2024-04-22T02:05:22.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 351 16 2502 2486 0.0553254 0 - 0.194117 2024-04-22T02:05:23.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 352 16 2502 2486 0.0551682 0 - 0.194117 2024-04-22T02:05:24.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 353 16 2502 2486 0.0550119 0 - 0.194117 2024-04-22T02:05:25.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 354 16 2502 2486 0.0548566 0 - 0.194117 2024-04-22T02:05:26.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 355 16 2502 2486 0.054702 0 - 0.194117 2024-04-22T02:05:27.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 356 16 2502 2486 0.0545484 0 - 0.194117 2024-04-22T02:05:28.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 357 16 2502 2486 0.0543956 0 - 0.194117 2024-04-22T02:05:29.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 358 16 2502 2486 0.0542436 0 - 0.194117 2024-04-22T02:05:30.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 359 16 2502 2486 0.0540925 0 - 0.194117 2024-04-22T02:05:31.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:05:31.280797+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:05:31.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:05:31.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 360 16 2502 2486 0.0539423 0 - 0.194117 2024-04-22T02:05:32.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 361 16 2502 2486 0.0537929 0 - 0.194117 2024-04-22T02:05:33.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 362 16 2502 2486 0.0536443 0 - 0.194117 2024-04-22T02:05:34.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 363 16 2502 2486 0.0534965 0 - 0.194117 2024-04-22T02:05:35.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 364 16 2502 2486 0.0533495 0 - 0.194117 2024-04-22T02:05:36.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 365 16 2502 2486 0.0532033 0 - 0.194117 2024-04-22T02:05:37.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 366 16 2502 2486 0.053058 0 - 0.194117 2024-04-22T02:05:38.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 367 16 2502 2486 0.0529134 0 - 0.194117 2024-04-22T02:05:39.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 368 16 2502 2486 0.0527696 0 - 0.194117 2024-04-22T02:05:40.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 369 16 2502 2486 0.0526266 0 - 0.194117 2024-04-22T02:05:41.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 370 16 2502 2486 0.0524844 0 - 0.194117 2024-04-22T02:05:42.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 371 16 2502 2486 0.0523429 0 - 0.194117 2024-04-22T02:05:43.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 372 16 2502 2486 0.0522022 0 - 0.194117 2024-04-22T02:05:44.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 373 16 2502 2486 0.0520623 0 - 0.194117 2024-04-22T02:05:45.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 374 16 2502 2486 0.051923 0 - 0.194117 2024-04-22T02:05:46.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 375 16 2502 2486 0.0517846 0 - 0.194117 2024-04-22T02:05:47.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 376 16 2502 2486 0.0516469 0 - 0.194117 2024-04-22T02:05:48.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 377 16 2502 2486 0.0515099 0 - 0.194117 2024-04-22T02:05:49.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 378 16 2502 2486 0.0513736 0 - 0.194117 2024-04-22T02:05:50.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 379 16 2502 2486 0.0512381 0 - 0.194117 2024-04-22T02:05:51.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:05:51.283481+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:05:51.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:05:51.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 380 16 2502 2486 0.0511032 0 - 0.194117 2024-04-22T02:05:52.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 381 16 2502 2486 0.0509691 0 - 0.194117 2024-04-22T02:05:53.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 382 16 2502 2486 0.0508357 0 - 0.194117 2024-04-22T02:05:54.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 383 16 2502 2486 0.0507029 0 - 0.194117 2024-04-22T02:05:55.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 384 16 2502 2486 0.0505709 0 - 0.194117 2024-04-22T02:05:56.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 385 16 2502 2486 0.0504396 0 - 0.194117 2024-04-22T02:05:57.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 386 16 2502 2486 0.0503089 0 - 0.194117 2024-04-22T02:05:58.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 387 16 2502 2486 0.0501789 0 - 0.194117 2024-04-22T02:05:59.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 388 16 2502 2486 0.0500496 0 - 0.194117 2024-04-22T02:06:00.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 389 16 2502 2486 0.0499209 0 - 0.194117 2024-04-22T02:06:01.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 390 16 2502 2486 0.0497929 0 - 0.194117 2024-04-22T02:06:02.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 391 16 2502 2486 0.0496656 0 - 0.194117 2024-04-22T02:06:03.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 392 16 2502 2486 0.0495389 0 - 0.194117 2024-04-22T02:06:04.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 393 16 2502 2486 0.0494128 0 - 0.194117 2024-04-22T02:06:05.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 394 16 2502 2486 0.0492874 0 - 0.194117 2024-04-22T02:06:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:06:05.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:06:06.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 395 16 2502 2486 0.0491626 0 - 0.194117 2024-04-22T02:06:07.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 396 16 2502 2486 0.0490385 0 - 0.194117 2024-04-22T02:06:07.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:06:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:06:07.547+0000 7efffa6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:06:08.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 397 16 2502 2486 0.048915 0 - 0.194117 2024-04-22T02:06:09.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 398 16 2502 2486 0.0487921 0 - 0.194117 2024-04-22T02:06:10.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 399 16 2502 2486 0.0486698 0 - 0.194117 2024-04-22T02:06:11.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:06:11.285787+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:06:11.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:06:11.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 400 16 2502 2486 0.0485481 0 - 0.194117 2024-04-22T02:06:12.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 401 16 2502 2486 0.048427 0 - 0.194117 2024-04-22T02:06:13.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 402 16 2502 2486 0.0483066 0 - 0.194117 2024-04-22T02:06:14.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 403 16 2502 2486 0.0481867 0 - 0.194117 2024-04-22T02:06:15.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 404 16 2502 2486 0.0480674 0 - 0.194117 2024-04-22T02:06:16.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 405 16 2502 2486 0.0479488 0 - 0.194117 2024-04-22T02:06:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:06:16.971 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:06:16.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-04-22T02:06:17.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:06:17.086+0000 7fc65cf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:06:17.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 406 16 2502 2486 0.0478307 0 - 0.194117 2024-04-22T02:06:18.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 407 16 2502 2486 0.0477132 0 - 0.194117 2024-04-22T02:06:19.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 408 16 2502 2486 0.0475962 0 - 0.194117 2024-04-22T02:06:20.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 409 16 2502 2486 0.0474798 0 - 0.194117 2024-04-22T02:06:21.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 410 16 2502 2486 0.047364 0 - 0.194117 2024-04-22T02:06:22.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 411 16 2502 2486 0.0472488 0 - 0.194117 2024-04-22T02:06:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 412 16 2502 2486 0.0471341 0 - 0.194117 2024-04-22T02:06:24.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 413 16 2502 2486 0.04702 0 - 0.194117 2024-04-22T02:06:25.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 414 16 2502 2486 0.0469064 0 - 0.194117 2024-04-22T02:06:26.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 415 16 2502 2486 0.0467934 0 - 0.194117 2024-04-22T02:06:27.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 416 16 2502 2486 0.0466809 0 - 0.194117 2024-04-22T02:06:28.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 417 16 2502 2486 0.046569 0 - 0.194117 2024-04-22T02:06:29.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 418 16 2502 2486 0.0464576 0 - 0.194117 2024-04-22T02:06:30.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 419 16 2502 2486 0.0463467 0 - 0.194117 2024-04-22T02:06:31.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:06:31.287874+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:06:31.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:06:31.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 420 16 2502 2486 0.0462364 0 - 0.194117 2024-04-22T02:06:32.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 421 16 2502 2486 0.0461265 0 - 0.194117 2024-04-22T02:06:33.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 422 16 2502 2486 0.0460172 0 - 0.194117 2024-04-22T02:06:34.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 423 16 2502 2486 0.0459085 0 - 0.194117 2024-04-22T02:06:35.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 424 16 2502 2486 0.0458002 0 - 0.194117 2024-04-22T02:06:36.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 425 16 2502 2486 0.0456924 0 - 0.194117 2024-04-22T02:06:37.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 426 16 2502 2486 0.0455852 0 - 0.194117 2024-04-22T02:06:38.286 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 427 16 2502 2486 0.0454784 0 - 0.194117 2024-04-22T02:06:39.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 428 16 2502 2486 0.0453722 0 - 0.194117 2024-04-22T02:06:40.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 429 16 2502 2486 0.0452664 0 - 0.194117 2024-04-22T02:06:41.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 430 16 2502 2486 0.0451611 0 - 0.194117 2024-04-22T02:06:42.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 431 16 2502 2486 0.0450564 0 - 0.194117 2024-04-22T02:06:43.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 432 16 2502 2486 0.0449521 0 - 0.194117 2024-04-22T02:06:44.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 433 16 2502 2486 0.0448483 0 - 0.194117 2024-04-22T02:06:45.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 434 16 2502 2486 0.0447449 0 - 0.194117 2024-04-22T02:06:46.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 435 16 2502 2486 0.0446421 0 - 0.194117 2024-04-22T02:06:47.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 436 16 2502 2486 0.0445397 0 - 0.194117 2024-04-22T02:06:48.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 437 16 2502 2486 0.0444378 0 - 0.194117 2024-04-22T02:06:49.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 438 16 2502 2486 0.0443363 0 - 0.194117 2024-04-22T02:06:50.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 439 16 2502 2486 0.0442353 0 - 0.194117 2024-04-22T02:06:51.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:06:51.289596+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:06:51.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:06:51.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 440 16 2502 2486 0.0441348 0 - 0.194117 2024-04-22T02:06:52.287 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 441 16 2502 2486 0.0440347 0 - 0.194117 2024-04-22T02:06:53.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 442 16 2502 2486 0.0439351 0 - 0.194117 2024-04-22T02:06:54.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 443 16 2502 2486 0.0438359 0 - 0.194117 2024-04-22T02:06:55.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 444 16 2502 2486 0.0437372 0 - 0.194117 2024-04-22T02:06:56.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 445 16 2502 2486 0.0436389 0 - 0.194117 2024-04-22T02:06:57.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 446 16 2502 2486 0.0435411 0 - 0.194117 2024-04-22T02:06:58.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 447 16 2502 2486 0.0434437 0 - 0.194117 2024-04-22T02:06:59.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 448 16 2502 2486 0.0433467 0 - 0.194117 2024-04-22T02:07:00.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 449 16 2502 2486 0.0432502 0 - 0.194117 2024-04-22T02:07:01.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 450 16 2502 2486 0.0431541 0 - 0.194117 2024-04-22T02:07:02.288 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 451 16 2502 2486 0.0430584 0 - 0.194117 2024-04-22T02:07:03.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 452 16 2502 2486 0.0429631 0 - 0.194117 2024-04-22T02:07:04.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 453 16 2502 2486 0.0428683 0 - 0.194117 2024-04-22T02:07:05.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 454 16 2502 2486 0.0427739 0 - 0.194117 2024-04-22T02:07:06.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 455 16 2502 2486 0.0426799 0 - 0.194117 2024-04-22T02:07:07.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 456 16 2502 2486 0.0425863 0 - 0.194117 2024-04-22T02:07:08.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 457 16 2502 2486 0.0424931 0 - 0.194117 2024-04-22T02:07:09.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 458 16 2502 2486 0.0424003 0 - 0.194117 2024-04-22T02:07:10.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 459 16 2502 2486 0.0423079 0 - 0.194117 2024-04-22T02:07:11.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:07:11.291496+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:07:11.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:07:11.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 460 16 2502 2486 0.042216 0 - 0.194117 2024-04-22T02:07:12.289 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 461 16 2502 2486 0.0421244 0 - 0.194117 2024-04-22T02:07:13.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 462 16 2502 2486 0.0420332 0 - 0.194117 2024-04-22T02:07:14.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 463 16 2502 2486 0.0419424 0 - 0.194117 2024-04-22T02:07:15.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 464 16 2502 2486 0.041852 0 - 0.194117 2024-04-22T02:07:16.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 465 16 2502 2486 0.041762 0 - 0.194117 2024-04-22T02:07:17.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 466 16 2502 2486 0.0416724 0 - 0.194117 2024-04-22T02:07:18.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 467 16 2502 2486 0.0415832 0 - 0.194117 2024-04-22T02:07:19.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 468 16 2502 2486 0.0414944 0 - 0.194117 2024-04-22T02:07:20.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 469 16 2502 2486 0.0414059 0 - 0.194117 2024-04-22T02:07:21.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 470 16 2502 2486 0.0413178 0 - 0.194117 2024-04-22T02:07:22.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 471 16 2502 2486 0.0412301 0 - 0.194117 2024-04-22T02:07:23.290 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 472 16 2502 2486 0.0411427 0 - 0.194117 2024-04-22T02:07:24.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 473 16 2502 2486 0.0410557 0 - 0.194117 2024-04-22T02:07:25.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 474 16 2502 2486 0.0409691 0 - 0.194117 2024-04-22T02:07:26.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 475 16 2502 2486 0.0408829 0 - 0.194117 2024-04-22T02:07:27.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 476 16 2502 2486 0.040797 0 - 0.194117 2024-04-22T02:07:28.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 477 16 2502 2486 0.0407115 0 - 0.194117 2024-04-22T02:07:29.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 478 16 2502 2486 0.0406263 0 - 0.194117 2024-04-22T02:07:30.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 479 16 2502 2486 0.0405415 0 - 0.194117 2024-04-22T02:07:31.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:07:31.293564+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:07:31.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:07:31.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 480 16 2502 2486 0.040457 0 - 0.194117 2024-04-22T02:07:32.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 481 16 2502 2486 0.0403729 0 - 0.194117 2024-04-22T02:07:33.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 482 16 2502 2486 0.0402891 0 - 0.194117 2024-04-22T02:07:34.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 483 16 2502 2486 0.0402057 0 - 0.194117 2024-04-22T02:07:35.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 484 16 2502 2486 0.0401227 0 - 0.194117 2024-04-22T02:07:36.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 485 16 2502 2486 0.0400399 0 - 0.194117 2024-04-22T02:07:37.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 486 16 2502 2486 0.0399576 0 - 0.194117 2024-04-22T02:07:38.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 487 16 2502 2486 0.0398755 0 - 0.194117 2024-04-22T02:07:39.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 488 16 2502 2486 0.0397938 0 - 0.194117 2024-04-22T02:07:40.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 489 16 2502 2486 0.0397124 0 - 0.194117 2024-04-22T02:07:41.292 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 490 16 2502 2486 0.0396314 0 - 0.194117 2024-04-22T02:07:42.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 491 16 2502 2486 0.0395507 0 - 0.194117 2024-04-22T02:07:43.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 492 16 2502 2486 0.0394703 0 - 0.194117 2024-04-22T02:07:44.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 493 16 2502 2486 0.0393902 0 - 0.194117 2024-04-22T02:07:45.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 494 16 2502 2486 0.0393105 0 - 0.194117 2024-04-22T02:07:46.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 495 16 2502 2486 0.0392311 0 - 0.194117 2024-04-22T02:07:47.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 496 16 2502 2486 0.039152 0 - 0.194117 2024-04-22T02:07:48.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 497 16 2502 2486 0.0390732 0 - 0.194117 2024-04-22T02:07:49.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 498 16 2502 2486 0.0389948 0 - 0.194117 2024-04-22T02:07:50.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 499 16 2502 2486 0.0389166 0 - 0.194117 2024-04-22T02:07:51.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:07:51.295564+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:07:51.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:07:51.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 500 16 2502 2486 0.0388388 0 - 0.194117 2024-04-22T02:07:52.293 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 501 16 2502 2486 0.0387613 0 - 0.194117 2024-04-22T02:07:53.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 502 16 2502 2486 0.038684 0 - 0.194117 2024-04-22T02:07:54.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 503 16 2502 2486 0.0386071 0 - 0.194117 2024-04-22T02:07:55.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 504 16 2502 2486 0.0385305 0 - 0.194117 2024-04-22T02:07:56.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 505 16 2502 2486 0.0384542 0 - 0.194117 2024-04-22T02:07:57.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 506 16 2502 2486 0.0383783 0 - 0.194117 2024-04-22T02:07:58.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 507 16 2502 2486 0.0383026 0 - 0.194117 2024-04-22T02:07:59.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 508 16 2502 2486 0.0382272 0 - 0.194117 2024-04-22T02:08:00.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 509 16 2502 2486 0.0381521 0 - 0.194117 2024-04-22T02:08:01.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 510 16 2502 2486 0.0380773 0 - 0.194117 2024-04-22T02:08:02.294 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 511 16 2502 2486 0.0380027 0 - 0.194117 2024-04-22T02:08:03.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 512 16 2502 2486 0.0379285 0 - 0.194117 2024-04-22T02:08:04.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 513 16 2502 2486 0.0378546 0 - 0.194117 2024-04-22T02:08:05.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 514 16 2502 2486 0.0377809 0 - 0.194117 2024-04-22T02:08:06.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 515 16 2502 2486 0.0377076 0 - 0.194117 2024-04-22T02:08:07.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 516 16 2502 2486 0.0376345 0 - 0.194117 2024-04-22T02:08:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:08:07.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:08:08.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 517 16 2502 2486 0.0375617 0 - 0.194117 2024-04-22T02:08:09.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 518 16 2502 2486 0.0374892 0 - 0.194117 2024-04-22T02:08:09.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:08:09.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:08:09.583+0000 7f636459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:08:10.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 519 16 2502 2486 0.037417 0 - 0.194117 2024-04-22T02:08:11.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:08:11.297598+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:08:11.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:08:11.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 520 16 2502 2486 0.037345 0 - 0.194117 2024-04-22T02:08:12.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 521 16 2502 2486 0.0372733 0 - 0.194117 2024-04-22T02:08:13.295 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 522 16 2502 2486 0.0372019 0 - 0.194117 2024-04-22T02:08:14.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 523 16 2502 2486 0.0371308 0 - 0.194117 2024-04-22T02:08:15.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 524 16 2502 2486 0.03706 0 - 0.194117 2024-04-22T02:08:16.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 525 16 2502 2486 0.0369894 0 - 0.194117 2024-04-22T02:08:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:08:17.005 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:08:17.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-04-22T02:08:17.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:08:17.119+0000 7f60cd59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:08:17.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 526 16 2502 2486 0.036919 0 - 0.194117 2024-04-22T02:08:18.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 527 16 2502 2486 0.036849 0 - 0.194117 2024-04-22T02:08:19.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 528 16 2502 2486 0.0367792 0 - 0.194117 2024-04-22T02:08:20.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 529 16 2502 2486 0.0367097 0 - 0.194117 2024-04-22T02:08:21.296 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 530 16 2502 2486 0.0366404 0 - 0.194117 2024-04-22T02:08:22.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 531 16 2502 2486 0.0365714 0 - 0.194117 2024-04-22T02:08:23.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 532 16 2502 2486 0.0365027 0 - 0.194117 2024-04-22T02:08:24.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 533 16 2502 2486 0.0364342 0 - 0.194117 2024-04-22T02:08:25.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 534 16 2502 2486 0.036366 0 - 0.194117 2024-04-22T02:08:26.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 535 16 2502 2486 0.036298 0 - 0.194117 2024-04-22T02:08:27.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 536 16 2502 2486 0.0362303 0 - 0.194117 2024-04-22T02:08:28.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 537 16 2502 2486 0.0361628 0 - 0.194117 2024-04-22T02:08:29.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 538 16 2502 2486 0.0360956 0 - 0.194117 2024-04-22T02:08:30.297 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 539 16 2502 2486 0.0360286 0 - 0.194117 2024-04-22T02:08:31.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:08:31.299676+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:08:31.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:08:31.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 540 16 2502 2486 0.0359619 0 - 0.194117 2024-04-22T02:08:32.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 541 16 2502 2486 0.0358954 0 - 0.194117 2024-04-22T02:08:33.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 542 16 2502 2486 0.0358292 0 - 0.194117 2024-04-22T02:08:34.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 543 16 2502 2486 0.0357632 0 - 0.194117 2024-04-22T02:08:35.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 544 16 2502 2486 0.0356975 0 - 0.194117 2024-04-22T02:08:36.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 545 16 2502 2486 0.035632 0 - 0.194117 2024-04-22T02:08:37.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 546 16 2502 2486 0.0355667 0 - 0.194117 2024-04-22T02:08:38.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 547 16 2502 2486 0.0355017 0 - 0.194117 2024-04-22T02:08:39.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 548 16 2502 2486 0.0354369 0 - 0.194117 2024-04-22T02:08:40.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 549 16 2502 2486 0.0353724 0 - 0.194117 2024-04-22T02:08:41.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 550 16 2502 2486 0.0353081 0 - 0.194117 2024-04-22T02:08:42.298 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 551 16 2502 2486 0.035244 0 - 0.194117 2024-04-22T02:08:43.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 552 16 2502 2486 0.0351802 0 - 0.194117 2024-04-22T02:08:44.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 553 16 2502 2486 0.0351165 0 - 0.194117 2024-04-22T02:08:45.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 554 16 2502 2486 0.0350532 0 - 0.194117 2024-04-22T02:08:46.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 555 16 2502 2486 0.03499 0 - 0.194117 2024-04-22T02:08:47.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 556 16 2502 2486 0.0349271 0 - 0.194117 2024-04-22T02:08:48.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 557 16 2502 2486 0.0348644 0 - 0.194117 2024-04-22T02:08:49.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 558 16 2502 2486 0.0348019 0 - 0.194117 2024-04-22T02:08:50.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 559 16 2502 2486 0.0347396 0 - 0.194117 2024-04-22T02:08:51.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:08:51.301453+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:08:51.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:08:51.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 560 16 2502 2486 0.0346776 0 - 0.194117 2024-04-22T02:08:52.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 561 16 2502 2486 0.0346158 0 - 0.194117 2024-04-22T02:08:53.299 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 562 16 2502 2486 0.0345542 0 - 0.194117 2024-04-22T02:08:54.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 563 16 2502 2486 0.0344928 0 - 0.194117 2024-04-22T02:08:55.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 564 16 2502 2486 0.0344317 0 - 0.194117 2024-04-22T02:08:56.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 565 16 2502 2486 0.0343707 0 - 0.194117 2024-04-22T02:08:57.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 566 16 2502 2486 0.03431 0 - 0.194117 2024-04-22T02:08:58.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 567 16 2502 2486 0.0342495 0 - 0.194117 2024-04-22T02:08:59.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 568 16 2502 2486 0.0341892 0 - 0.194117 2024-04-22T02:09:00.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 569 16 2502 2486 0.0341291 0 - 0.194117 2024-04-22T02:09:01.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 570 16 2502 2486 0.0340692 0 - 0.194117 2024-04-22T02:09:02.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 571 16 2502 2486 0.0340096 0 - 0.194117 2024-04-22T02:09:03.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 572 16 2502 2486 0.0339501 0 - 0.194117 2024-04-22T02:09:04.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 573 16 2502 2486 0.0338909 0 - 0.194117 2024-04-22T02:09:05.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 574 16 2502 2486 0.0338318 0 - 0.194117 2024-04-22T02:09:06.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 575 16 2502 2486 0.033773 0 - 0.194117 2024-04-22T02:09:07.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 576 16 2502 2486 0.0337144 0 - 0.194117 2024-04-22T02:09:08.300 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 577 16 2502 2486 0.0336559 0 - 0.194117 2024-04-22T02:09:09.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 578 16 2502 2486 0.0335977 0 - 0.194117 2024-04-22T02:09:10.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 579 16 2502 2486 0.0335397 0 - 0.194117 2024-04-22T02:09:11.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:09:11.303070+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:09:11.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:09:11.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 580 16 2502 2486 0.0334819 0 - 0.194117 2024-04-22T02:09:12.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 581 16 2502 2486 0.0334242 0 - 0.194117 2024-04-22T02:09:13.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 582 16 2502 2486 0.0333668 0 - 0.194117 2024-04-22T02:09:14.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 583 16 2502 2486 0.0333096 0 - 0.194117 2024-04-22T02:09:15.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 584 16 2502 2486 0.0332525 0 - 0.194117 2024-04-22T02:09:16.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 585 16 2502 2486 0.0331957 0 - 0.194117 2024-04-22T02:09:17.301 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 586 16 2502 2486 0.0331391 0 - 0.194117 2024-04-22T02:09:18.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 587 16 2502 2486 0.0330826 0 - 0.194117 2024-04-22T02:09:19.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 588 16 2502 2486 0.0330263 0 - 0.194117 2024-04-22T02:09:20.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 589 16 2502 2486 0.0329703 0 - 0.194117 2024-04-22T02:09:21.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 590 16 2502 2486 0.0329144 0 - 0.194117 2024-04-22T02:09:22.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 591 16 2502 2486 0.0328587 0 - 0.194117 2024-04-22T02:09:23.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 592 16 2502 2486 0.0328032 0 - 0.194117 2024-04-22T02:09:24.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 593 16 2502 2486 0.0327479 0 - 0.194117 2024-04-22T02:09:25.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 594 16 2502 2486 0.0326927 0 - 0.194117 2024-04-22T02:09:26.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 595 16 2502 2486 0.0326378 0 - 0.194117 2024-04-22T02:09:27.302 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 596 16 2502 2486 0.032583 0 - 0.194117 2024-04-22T02:09:28.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 597 16 2502 2486 0.0325285 0 - 0.194117 2024-04-22T02:09:29.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 598 16 2502 2486 0.0324741 0 - 0.194117 2024-04-22T02:09:30.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 599 16 2502 2486 0.0324199 0 - 0.194117 2024-04-22T02:09:31.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:09:31.305052+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:09:31.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:09:31.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 600 16 2502 2486 0.0323658 0 - 0.194117 2024-04-22T02:09:32.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 601 16 2502 2486 0.032312 0 - 0.194117 2024-04-22T02:09:33.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 602 16 2502 2486 0.0322583 0 - 0.194117 2024-04-22T02:09:34.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 603 16 2502 2486 0.0322048 0 - 0.194117 2024-04-22T02:09:35.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 604 16 2502 2486 0.0321515 0 - 0.194117 2024-04-22T02:09:36.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 605 16 2502 2486 0.0320984 0 - 0.194117 2024-04-22T02:09:37.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 606 16 2502 2486 0.0320454 0 - 0.194117 2024-04-22T02:09:38.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 607 16 2502 2486 0.0319926 0 - 0.194117 2024-04-22T02:09:39.303 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 608 16 2502 2486 0.03194 0 - 0.194117 2024-04-22T02:09:40.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 609 16 2502 2486 0.0318875 0 - 0.194117 2024-04-22T02:09:41.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 610 16 2502 2486 0.0318353 0 - 0.194117 2024-04-22T02:09:42.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 611 16 2502 2486 0.0317832 0 - 0.194117 2024-04-22T02:09:43.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 612 16 2502 2486 0.0317312 0 - 0.194117 2024-04-22T02:09:44.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 613 16 2502 2486 0.0316795 0 - 0.194117 2024-04-22T02:09:45.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 614 16 2502 2486 0.0316279 0 - 0.194117 2024-04-22T02:09:46.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 615 16 2502 2486 0.0315764 0 - 0.194117 2024-04-22T02:09:47.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 616 16 2502 2486 0.0315252 0 - 0.194117 2024-04-22T02:09:48.304 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 617 16 2502 2486 0.0314741 0 - 0.194117 2024-04-22T02:09:49.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 618 16 2502 2486 0.0314232 0 - 0.194117 2024-04-22T02:09:50.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 619 16 2502 2486 0.0313724 0 - 0.194117 2024-04-22T02:09:51.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:09:51.306910+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:09:51.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:09:51.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 620 16 2502 2486 0.0313218 0 - 0.194117 2024-04-22T02:09:52.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 621 16 2502 2486 0.0312714 0 - 0.194117 2024-04-22T02:09:53.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 622 16 2502 2486 0.0312211 0 - 0.194117 2024-04-22T02:09:54.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 623 16 2502 2486 0.031171 0 - 0.194117 2024-04-22T02:09:55.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 624 16 2502 2486 0.031121 0 - 0.194117 2024-04-22T02:09:56.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 625 16 2502 2486 0.0310712 0 - 0.194117 2024-04-22T02:09:57.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 626 16 2502 2486 0.0310216 0 - 0.194117 2024-04-22T02:09:58.305 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 627 16 2502 2486 0.0309721 0 - 0.194117 2024-04-22T02:09:59.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 628 16 2502 2486 0.0309228 0 - 0.194117 2024-04-22T02:10:00.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 629 16 2502 2486 0.0308736 0 - 0.194117 2024-04-22T02:10:01.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 630 16 2502 2486 0.0308246 0 - 0.194117 2024-04-22T02:10:02.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 631 16 2502 2486 0.0307758 0 - 0.194117 2024-04-22T02:10:03.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 632 16 2502 2486 0.0307271 0 - 0.194117 2024-04-22T02:10:04.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 633 16 2502 2486 0.0306786 0 - 0.194117 2024-04-22T02:10:05.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 634 16 2502 2486 0.0306302 0 - 0.194117 2024-04-22T02:10:06.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 635 16 2502 2486 0.0305819 0 - 0.194117 2024-04-22T02:10:07.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 636 16 2502 2486 0.0305339 0 - 0.194117 2024-04-22T02:10:08.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 637 16 2502 2486 0.0304859 0 - 0.194117 2024-04-22T02:10:09.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 638 16 2502 2486 0.0304381 0 - 0.194117 2024-04-22T02:10:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:10:09.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:10:10.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 639 16 2502 2486 0.0303905 0 - 0.194117 2024-04-22T02:10:11.306 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:10:11.308733+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:10:11.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:10:11.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 640 16 2502 2486 0.030343 0 - 0.194117 2024-04-22T02:10:11.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:10:11.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:10:11.619+0000 7fd8e886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:10:12.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 641 16 2502 2486 0.0302957 0 - 0.194117 2024-04-22T02:10:13.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 642 16 2502 2486 0.0302485 0 - 0.194117 2024-04-22T02:10:14.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 643 16 2502 2486 0.0302015 0 - 0.194117 2024-04-22T02:10:15.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 644 16 2502 2486 0.0301546 0 - 0.194117 2024-04-22T02:10:16.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 645 16 2502 2486 0.0301078 0 - 0.194117 2024-04-22T02:10:17.038 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:10:17.039 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:10:17.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-04-22T02:10:17.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:10:17.153+0000 7f780a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:10:17.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 646 16 2502 2486 0.0300612 0 - 0.194117 2024-04-22T02:10:18.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 647 16 2502 2486 0.0300147 0 - 0.194117 2024-04-22T02:10:19.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 648 16 2502 2486 0.0299684 0 - 0.194117 2024-04-22T02:10:20.307 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 649 16 2502 2486 0.0299222 0 - 0.194117 2024-04-22T02:10:21.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 650 16 2502 2486 0.0298762 0 - 0.194117 2024-04-22T02:10:22.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 651 16 2502 2486 0.0298303 0 - 0.194117 2024-04-22T02:10:23.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 652 16 2502 2486 0.0297846 0 - 0.194117 2024-04-22T02:10:24.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 653 16 2502 2486 0.029739 0 - 0.194117 2024-04-22T02:10:25.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 654 16 2502 2486 0.0296935 0 - 0.194117 2024-04-22T02:10:26.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 655 16 2502 2486 0.0296481 0 - 0.194117 2024-04-22T02:10:27.308 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 656 16 2502 2486 0.029603 0 - 0.194117 2024-04-22T02:10:28.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 657 16 2502 2486 0.0295579 0 - 0.194117 2024-04-22T02:10:29.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 658 16 2502 2486 0.029513 0 - 0.194117 2024-04-22T02:10:30.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 659 16 2502 2486 0.0294682 0 - 0.194117 2024-04-22T02:10:31.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:10:31.311032+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:10:31.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:10:31.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 660 16 2502 2486 0.0294235 0 - 0.194117 2024-04-22T02:10:32.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 661 16 2502 2486 0.029379 0 - 0.194117 2024-04-22T02:10:33.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 662 16 2502 2486 0.0293346 0 - 0.194117 2024-04-22T02:10:34.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 663 16 2502 2486 0.0292904 0 - 0.194117 2024-04-22T02:10:35.309 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 664 16 2502 2486 0.0292463 0 - 0.194117 2024-04-22T02:10:36.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 665 16 2502 2486 0.0292023 0 - 0.194117 2024-04-22T02:10:37.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 666 16 2502 2486 0.0291585 0 - 0.194117 2024-04-22T02:10:38.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 667 16 2502 2486 0.0291147 0 - 0.194117 2024-04-22T02:10:39.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 668 16 2502 2486 0.0290712 0 - 0.194117 2024-04-22T02:10:40.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 669 16 2502 2486 0.0290277 0 - 0.194117 2024-04-22T02:10:41.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 670 16 2502 2486 0.0289844 0 - 0.194117 2024-04-22T02:10:42.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 671 16 2502 2486 0.0289412 0 - 0.194117 2024-04-22T02:10:43.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 672 16 2502 2486 0.0288981 0 - 0.194117 2024-04-22T02:10:44.310 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 673 16 2502 2486 0.0288552 0 - 0.194117 2024-04-22T02:10:45.311 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 674 16 2502 2486 0.0288124 0 - 0.194117 2024-04-22T02:10:46.311 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 675 16 2502 2486 0.0287697 0 - 0.194117 2024-04-22T02:10:47.311 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 676 16 2502 2486 0.0287271 0 - 0.194117 2024-04-22T02:10:48.311 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 677 16 2502 2486 0.0286847 0 - 0.194117 2024-04-22T02:10:49.311 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 678 16 2502 2486 0.0286424 0 - 0.194117 2024-04-22T02:10:50.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 679 16 2502 2486 0.0286002 0 - 0.194117 2024-04-22T02:10:51.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:10:51.313589+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:10:51.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:10:51.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 680 16 2502 2486 0.0285581 0 - 0.194117 2024-04-22T02:10:52.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 681 16 2502 2486 0.0285162 0 - 0.194117 2024-04-22T02:10:53.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 682 16 2502 2486 0.0284744 0 - 0.194117 2024-04-22T02:10:54.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 683 16 2502 2486 0.0284327 0 - 0.194117 2024-04-22T02:10:55.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 684 16 2502 2486 0.0283911 0 - 0.194117 2024-04-22T02:10:56.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 685 16 2502 2486 0.0283497 0 - 0.194117 2024-04-22T02:10:57.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 686 16 2502 2486 0.0283084 0 - 0.194117 2024-04-22T02:10:58.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 687 16 2502 2486 0.0282672 0 - 0.194117 2024-04-22T02:10:59.312 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 688 16 2502 2486 0.0282261 0 - 0.194117 2024-04-22T02:11:00.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 689 16 2502 2486 0.0281851 0 - 0.194117 2024-04-22T02:11:01.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 690 16 2502 2486 0.0281443 0 - 0.194117 2024-04-22T02:11:02.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 691 16 2502 2486 0.0281035 0 - 0.194117 2024-04-22T02:11:03.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 692 16 2502 2486 0.0280629 0 - 0.194117 2024-04-22T02:11:04.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 693 16 2502 2486 0.0280224 0 - 0.194117 2024-04-22T02:11:05.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 694 16 2502 2486 0.027982 0 - 0.194117 2024-04-22T02:11:06.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 695 16 2502 2486 0.0279418 0 - 0.194117 2024-04-22T02:11:07.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 696 16 2502 2486 0.0279016 0 - 0.194117 2024-04-22T02:11:08.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 697 16 2502 2486 0.0278616 0 - 0.194117 2024-04-22T02:11:09.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 698 16 2502 2486 0.0278217 0 - 0.194117 2024-04-22T02:11:10.313 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 699 16 2502 2486 0.0277819 0 - 0.194117 2024-04-22T02:11:11.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:11:11.315655+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:11:11.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:11:11.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 700 16 2502 2486 0.0277422 0 - 0.194117 2024-04-22T02:11:12.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 701 16 2502 2486 0.0277026 0 - 0.194117 2024-04-22T02:11:13.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 702 16 2502 2486 0.0276632 0 - 0.194117 2024-04-22T02:11:14.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 703 16 2502 2486 0.0276238 0 - 0.194117 2024-04-22T02:11:15.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 704 16 2502 2486 0.0275846 0 - 0.194117 2024-04-22T02:11:16.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 705 16 2502 2486 0.0275455 0 - 0.194117 2024-04-22T02:11:17.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 706 16 2502 2486 0.0275064 0 - 0.194117 2024-04-22T02:11:18.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 707 16 2502 2486 0.0274675 0 - 0.194117 2024-04-22T02:11:19.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 708 16 2502 2486 0.0274287 0 - 0.194117 2024-04-22T02:11:20.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 709 16 2502 2486 0.0273901 0 - 0.194117 2024-04-22T02:11:21.314 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 710 16 2502 2486 0.0273515 0 - 0.194117 2024-04-22T02:11:22.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 711 16 2502 2486 0.027313 0 - 0.194117 2024-04-22T02:11:23.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 712 16 2502 2486 0.0272746 0 - 0.194117 2024-04-22T02:11:24.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 713 16 2502 2486 0.0272364 0 - 0.194117 2024-04-22T02:11:25.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 714 16 2502 2486 0.0271982 0 - 0.194117 2024-04-22T02:11:26.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 715 16 2502 2486 0.0271602 0 - 0.194117 2024-04-22T02:11:27.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 716 16 2502 2486 0.0271223 0 - 0.194117 2024-04-22T02:11:28.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 717 16 2502 2486 0.0270845 0 - 0.194117 2024-04-22T02:11:29.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 718 16 2502 2486 0.0270467 0 - 0.194117 2024-04-22T02:11:30.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 719 16 2502 2486 0.0270091 0 - 0.194117 2024-04-22T02:11:31.315 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:11:31.317661+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:11:31.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:11:31.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 720 16 2502 2486 0.0269716 0 - 0.194117 2024-04-22T02:11:32.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 721 16 2502 2486 0.0269342 0 - 0.194117 2024-04-22T02:11:33.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 722 16 2502 2486 0.0268969 0 - 0.194117 2024-04-22T02:11:34.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 723 16 2502 2486 0.0268597 0 - 0.194117 2024-04-22T02:11:35.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 724 16 2502 2486 0.0268226 0 - 0.194117 2024-04-22T02:11:36.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 725 16 2502 2486 0.0267856 0 - 0.194117 2024-04-22T02:11:37.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 726 16 2502 2486 0.0267487 0 - 0.194117 2024-04-22T02:11:38.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 727 16 2502 2486 0.0267119 0 - 0.194117 2024-04-22T02:11:39.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 728 16 2502 2486 0.0266752 0 - 0.194117 2024-04-22T02:11:40.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 729 16 2502 2486 0.0266386 0 - 0.194117 2024-04-22T02:11:41.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 730 16 2502 2486 0.0266021 0 - 0.194117 2024-04-22T02:11:42.316 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 731 16 2502 2486 0.0265657 0 - 0.194117 2024-04-22T02:11:43.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 732 16 2502 2486 0.0265295 0 - 0.194117 2024-04-22T02:11:44.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 733 16 2502 2486 0.0264933 0 - 0.194117 2024-04-22T02:11:45.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 734 16 2502 2486 0.0264572 0 - 0.194117 2024-04-22T02:11:46.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 735 16 2502 2486 0.0264212 0 - 0.194117 2024-04-22T02:11:47.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 736 16 2502 2486 0.0263853 0 - 0.194117 2024-04-22T02:11:48.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 737 16 2502 2486 0.0263495 0 - 0.194117 2024-04-22T02:11:49.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 738 16 2502 2486 0.0263138 0 - 0.194117 2024-04-22T02:11:50.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 739 16 2502 2486 0.0262782 0 - 0.194117 2024-04-22T02:11:51.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:11:51.319570+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:11:51.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:11:51.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 740 16 2502 2486 0.0262427 0 - 0.194117 2024-04-22T02:11:52.317 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 741 16 2502 2486 0.0262072 0 - 0.194117 2024-04-22T02:11:53.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 742 16 2502 2486 0.0261719 0 - 0.194117 2024-04-22T02:11:54.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 743 16 2502 2486 0.0261367 0 - 0.194117 2024-04-22T02:11:55.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 744 16 2502 2486 0.0261016 0 - 0.194117 2024-04-22T02:11:56.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 745 16 2502 2486 0.0260665 0 - 0.194117 2024-04-22T02:11:57.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 746 16 2502 2486 0.0260316 0 - 0.194117 2024-04-22T02:11:58.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 747 16 2502 2486 0.0259967 0 - 0.194117 2024-04-22T02:11:59.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 748 16 2502 2486 0.025962 0 - 0.194117 2024-04-22T02:12:00.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 749 16 2502 2486 0.0259273 0 - 0.194117 2024-04-22T02:12:01.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 750 16 2502 2486 0.0258928 0 - 0.194117 2024-04-22T02:12:02.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 751 16 2502 2486 0.0258583 0 - 0.194117 2024-04-22T02:12:03.318 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 752 16 2502 2486 0.0258239 0 - 0.194117 2024-04-22T02:12:04.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 753 16 2502 2486 0.0257896 0 - 0.194117 2024-04-22T02:12:05.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 754 16 2502 2486 0.0257554 0 - 0.194117 2024-04-22T02:12:06.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 755 16 2502 2486 0.0257213 0 - 0.194117 2024-04-22T02:12:07.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 756 16 2502 2486 0.0256873 0 - 0.194117 2024-04-22T02:12:08.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 757 16 2502 2486 0.0256533 0 - 0.194117 2024-04-22T02:12:09.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 758 16 2502 2486 0.0256195 0 - 0.194117 2024-04-22T02:12:10.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 759 16 2502 2486 0.0255857 0 - 0.194117 2024-04-22T02:12:11.319 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:12:11.321555+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:12:11.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:12:11.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 760 16 2502 2486 0.0255521 0 - 0.194117 2024-04-22T02:12:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:12:11.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:12:12.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 761 16 2502 2486 0.0255185 0 - 0.194117 2024-04-22T02:12:13.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 762 16 2502 2486 0.025485 0 - 0.194117 2024-04-22T02:12:13.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:12:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:12:13.655+0000 7f12c1664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:12:14.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 763 16 2502 2486 0.0254516 0 - 0.194117 2024-04-22T02:12:15.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 764 16 2502 2486 0.0254183 0 - 0.194117 2024-04-22T02:12:16.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 765 16 2502 2486 0.0253851 0 - 0.194117 2024-04-22T02:12:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:12:17.073 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-04-22T02:12:17.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 766 16 2502 2486 0.0253519 0 - 0.194117 2024-04-22T02:12:18.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 767 16 2502 2486 0.0253189 0 - 0.194117 2024-04-22T02:12:19.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 768 16 2502 2486 0.0252859 0 - 0.194117 2024-04-22T02:12:20.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 769 16 2502 2486 0.025253 0 - 0.194117 2024-04-22T02:12:21.320 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 770 16 2502 2486 0.0252202 0 - 0.194117 2024-04-22T02:12:22.074 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2, 3] live_osds: [1, 0] 2024-04-22T02:12:22.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-22T02:12:22.074 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-04-22T02:12:22.075 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-22T02:12:22.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-04-22T02:12:22.078 INFO:tasks.ceph.osd.3:Started 2024-04-22T02:12:22.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:22.189 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:22.193 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:22.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 771 16 2502 2486 0.0251875 0 - 0.194117 2024-04-22T02:12:23.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 772 16 2502 2486 0.0251549 0 - 0.194117 2024-04-22T02:12:24.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 773 16 2502 2486 0.0251224 0 - 0.194117 2024-04-22T02:12:25.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 774 16 2502 2486 0.0250899 0 - 0.194117 2024-04-22T02:12:26.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 775 16 2502 2486 0.0250575 0 - 0.194117 2024-04-22T02:12:27.194 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:27.303 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:27.306 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:27.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 776 16 2502 2486 0.0250252 0 - 0.194117 2024-04-22T02:12:28.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 777 16 2502 2486 0.024993 0 - 0.194117 2024-04-22T02:12:29.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 778 16 2502 2486 0.0249609 0 - 0.194117 2024-04-22T02:12:30.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 779 16 2502 2486 0.0249289 0 - 0.194117 2024-04-22T02:12:31.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:12:31.323505+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:12:31.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:12:31.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 780 16 2502 2486 0.0248969 0 - 0.194117 2024-04-22T02:12:32.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:32.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 781 16 2502 2486 0.024865 0 - 0.194117 2024-04-22T02:12:32.415 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:32.419 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:33.321 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 782 16 2502 2486 0.0248332 0 - 0.194117 2024-04-22T02:12:34.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 783 16 2502 2486 0.0248015 0 - 0.194117 2024-04-22T02:12:35.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 784 16 2502 2486 0.0247699 0 - 0.194117 2024-04-22T02:12:36.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 785 16 2502 2486 0.0247383 0 - 0.194117 2024-04-22T02:12:37.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 786 16 2502 2486 0.0247069 0 - 0.194117 2024-04-22T02:12:37.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:37.529 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:37.532 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:38.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 787 16 2502 2486 0.0246755 0 - 0.194117 2024-04-22T02:12:39.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 788 16 2502 2486 0.0246442 0 - 0.194117 2024-04-22T02:12:40.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 789 16 2502 2486 0.0246129 0 - 0.194117 2024-04-22T02:12:41.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 790 16 2502 2486 0.0245818 0 - 0.194117 2024-04-22T02:12:42.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 791 16 2502 2486 0.0245507 0 - 0.194117 2024-04-22T02:12:42.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:42.641 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:42.645 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:43.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 792 16 2502 2486 0.0245197 0 - 0.194117 2024-04-22T02:12:44.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 793 16 2502 2486 0.0244888 0 - 0.194117 2024-04-22T02:12:45.322 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 794 16 2502 2486 0.0244579 0 - 0.194117 2024-04-22T02:12:46.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 795 16 2502 2486 0.0244272 0 - 0.194117 2024-04-22T02:12:47.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 796 16 2502 2486 0.0243965 0 - 0.194117 2024-04-22T02:12:47.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:47.755 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:47.758 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:48.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 797 16 2502 2486 0.0243659 0 - 0.194117 2024-04-22T02:12:49.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 798 16 2502 2486 0.0243353 0 - 0.194117 2024-04-22T02:12:50.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 799 16 2502 2486 0.0243049 0 - 0.194117 2024-04-22T02:12:51.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:12:51.325279+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:12:51.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:12:51.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 800 16 2502 2486 0.0242745 0 - 0.194117 2024-04-22T02:12:52.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 801 16 2502 2486 0.0242442 0 - 0.194117 2024-04-22T02:12:52.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:52.867 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:52.871 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:53.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 802 16 2502 2486 0.024214 0 - 0.194117 2024-04-22T02:12:54.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 803 16 2502 2486 0.0241838 0 - 0.194117 2024-04-22T02:12:55.323 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 804 16 2502 2486 0.0241537 0 - 0.194117 2024-04-22T02:12:56.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 805 16 2502 2486 0.0241237 0 - 0.194117 2024-04-22T02:12:57.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 806 16 2502 2486 0.0240938 0 - 0.194117 2024-04-22T02:12:57.872 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:12:57.982 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:12:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:12:57.984 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:12:58.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 807 16 2502 2486 0.0240639 0 - 0.194117 2024-04-22T02:12:59.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 808 16 2502 2486 0.0240342 0 - 0.194117 2024-04-22T02:13:00.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 809 16 2502 2486 0.0240045 0 - 0.194117 2024-04-22T02:13:01.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 810 16 2502 2486 0.0239748 0 - 0.194117 2024-04-22T02:13:02.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 811 16 2502 2486 0.0239453 0 - 0.194117 2024-04-22T02:13:02.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:03.096 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:03.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 812 16 2502 2486 0.0239158 0 - 0.194117 2024-04-22T02:13:04.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 813 16 2502 2486 0.0238864 0 - 0.194117 2024-04-22T02:13:05.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 814 16 2502 2486 0.023857 0 - 0.194117 2024-04-22T02:13:06.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 815 16 2502 2486 0.0238277 0 - 0.194117 2024-04-22T02:13:07.324 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 816 16 2502 2486 0.0237985 0 - 0.194117 2024-04-22T02:13:08.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:08.206 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:08.209 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:08.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 817 16 2502 2486 0.0237694 0 - 0.194117 2024-04-22T02:13:09.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 818 16 2502 2486 0.0237404 0 - 0.194117 2024-04-22T02:13:10.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 819 16 2502 2486 0.0237114 0 - 0.194117 2024-04-22T02:13:11.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:13:11.327035+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:13:11.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:13:11.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 820 16 2502 2486 0.0236825 0 - 0.194117 2024-04-22T02:13:12.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 821 16 2502 2486 0.0236536 0 - 0.194117 2024-04-22T02:13:13.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:13.318 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:13.321 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:13.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 822 16 2502 2486 0.0236248 0 - 0.194117 2024-04-22T02:13:14.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 823 16 2502 2486 0.0235961 0 - 0.194117 2024-04-22T02:13:15.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 824 16 2502 2486 0.0235675 0 - 0.194117 2024-04-22T02:13:16.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 825 16 2502 2486 0.0235389 0 - 0.194117 2024-04-22T02:13:17.325 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 826 16 2502 2486 0.0235104 0 - 0.194117 2024-04-22T02:13:18.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:18.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 827 16 2502 2486 0.023482 0 - 0.194117 2024-04-22T02:13:18.431 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:18.439 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:19.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 828 16 2502 2486 0.0234536 0 - 0.194117 2024-04-22T02:13:20.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 829 16 2502 2486 0.0234254 0 - 0.194117 2024-04-22T02:13:21.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 830 16 2502 2486 0.0233971 0 - 0.194117 2024-04-22T02:13:22.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 831 16 2502 2486 0.023369 0 - 0.194117 2024-04-22T02:13:23.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 832 16 2502 2486 0.0233409 0 - 0.194117 2024-04-22T02:13:23.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:23.548 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:23.551 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:24.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 833 16 2502 2486 0.0233129 0 - 0.194117 2024-04-22T02:13:25.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 834 16 2502 2486 0.0232849 0 - 0.194117 2024-04-22T02:13:26.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 835 16 2502 2486 0.023257 0 - 0.194117 2024-04-22T02:13:27.326 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 836 16 2502 2486 0.0232292 0 - 0.194117 2024-04-22T02:13:28.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 837 16 2502 2486 0.0232015 0 - 0.194117 2024-04-22T02:13:28.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:28.659 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:28.663 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:29.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 838 16 2502 2486 0.0231738 0 - 0.194117 2024-04-22T02:13:30.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 839 16 2502 2486 0.0231462 0 - 0.194117 2024-04-22T02:13:31.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:13:31.329189+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:13:31.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:13:31.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 840 16 2502 2486 0.0231186 0 - 0.194117 2024-04-22T02:13:32.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 841 16 2502 2486 0.0230911 0 - 0.194117 2024-04-22T02:13:33.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 842 16 2502 2486 0.0230637 0 - 0.194117 2024-04-22T02:13:33.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:33.771 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:33.774 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:34.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 843 16 2502 2486 0.0230363 0 - 0.194117 2024-04-22T02:13:35.327 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 844 16 2502 2486 0.023009 0 - 0.194117 2024-04-22T02:13:36.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 845 16 2502 2486 0.0229818 0 - 0.194117 2024-04-22T02:13:37.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 846 16 2502 2486 0.0229546 0 - 0.194117 2024-04-22T02:13:38.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 847 16 2502 2486 0.0229275 0 - 0.194117 2024-04-22T02:13:38.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:38.882 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:38.886 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:39.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 848 16 2502 2486 0.0229005 0 - 0.194117 2024-04-22T02:13:40.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 849 16 2502 2486 0.0228735 0 - 0.194117 2024-04-22T02:13:41.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 850 16 2502 2486 0.0228466 0 - 0.194117 2024-04-22T02:13:42.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 851 16 2502 2486 0.0228198 0 - 0.194117 2024-04-22T02:13:43.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 852 16 2502 2486 0.022793 0 - 0.194117 2024-04-22T02:13:43.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:43.994 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:43.997 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:44.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 853 16 2502 2486 0.0227663 0 - 0.194117 2024-04-22T02:13:45.328 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 854 16 2502 2486 0.0227396 0 - 0.194117 2024-04-22T02:13:46.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 855 16 2502 2486 0.022713 0 - 0.194117 2024-04-22T02:13:47.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 856 16 2502 2486 0.0226865 0 - 0.194117 2024-04-22T02:13:48.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 857 16 2502 2486 0.02266 0 - 0.194117 2024-04-22T02:13:48.998 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:49.106 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:49.110 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:49.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 858 16 2502 2486 0.0226336 0 - 0.194117 2024-04-22T02:13:50.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 859 16 2502 2486 0.0226072 0 - 0.194117 2024-04-22T02:13:51.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:13:51.331263+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:13:51.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:13:51.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 860 16 2502 2486 0.022581 0 - 0.194117 2024-04-22T02:13:52.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 861 16 2502 2486 0.0225547 0 - 0.194117 2024-04-22T02:13:53.329 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 862 16 2502 2486 0.0225286 0 - 0.194117 2024-04-22T02:13:54.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:54.218 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:54.222 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:13:54.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 863 16 2502 2486 0.0225025 0 - 0.194117 2024-04-22T02:13:55.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 864 16 2502 2486 0.0224764 0 - 0.194117 2024-04-22T02:13:56.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 865 16 2502 2486 0.0224504 0 - 0.194117 2024-04-22T02:13:57.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 866 16 2502 2486 0.0224245 0 - 0.194117 2024-04-22T02:13:58.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 867 16 2502 2486 0.0223986 0 - 0.194117 2024-04-22T02:13:59.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:13:59.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 868 16 2502 2486 0.0223728 0 - 0.194117 2024-04-22T02:13:59.333 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:13:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:13:59.336 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:00.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 869 16 2502 2486 0.0223471 0 - 0.194117 2024-04-22T02:14:01.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 870 16 2502 2486 0.0223214 0 - 0.194117 2024-04-22T02:14:02.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 871 16 2502 2486 0.0222958 0 - 0.194117 2024-04-22T02:14:03.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 872 16 2502 2486 0.0222702 0 - 0.194117 2024-04-22T02:14:04.330 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 873 16 2502 2486 0.0222447 0 - 0.194117 2024-04-22T02:14:04.337 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:04.448 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:04.451 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:05.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 874 16 2502 2486 0.0222193 0 - 0.194117 2024-04-22T02:14:06.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 875 16 2502 2486 0.0221939 0 - 0.194117 2024-04-22T02:14:07.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 876 16 2502 2486 0.0221685 0 - 0.194117 2024-04-22T02:14:08.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 877 16 2502 2486 0.0221432 0 - 0.194117 2024-04-22T02:14:09.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 878 16 2502 2486 0.022118 0 - 0.194117 2024-04-22T02:14:09.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:09.560 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:09.564 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:10.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 879 16 2502 2486 0.0220929 0 - 0.194117 2024-04-22T02:14:11.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:14:11.333302+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:14:11.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:14:11.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 880 16 2502 2486 0.0220678 0 - 0.194117 2024-04-22T02:14:12.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 881 16 2502 2486 0.0220427 0 - 0.194117 2024-04-22T02:14:13.331 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 882 16 2502 2486 0.0220177 0 - 0.194117 2024-04-22T02:14:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:14:13.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:14:14.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 883 16 2502 2486 0.0219928 0 - 0.194117 2024-04-22T02:14:14.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:14.672 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:14.675 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:15.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 884 16 2502 2486 0.0219679 0 - 0.194117 2024-04-22T02:14:15.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:14:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:14:15.690+0000 7f5bbbeed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:14:16.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 885 16 2502 2486 0.0219431 0 - 0.194117 2024-04-22T02:14:17.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 886 16 2502 2486 0.0219183 0 - 0.194117 2024-04-22T02:14:18.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 887 16 2502 2486 0.0218936 0 - 0.194117 2024-04-22T02:14:19.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 888 16 2502 2486 0.021869 0 - 0.194117 2024-04-22T02:14:19.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:19.785 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:19.788 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:20.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 889 16 2502 2486 0.0218444 0 - 0.194117 2024-04-22T02:14:21.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 890 16 2502 2486 0.0218198 0 - 0.194117 2024-04-22T02:14:22.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 891 16 2502 2486 0.0217953 0 - 0.194117 2024-04-22T02:14:23.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 892 16 2502 2486 0.0217709 0 - 0.194117 2024-04-22T02:14:24.332 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 893 16 2502 2486 0.0217465 0 - 0.194117 2024-04-22T02:14:24.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:24.897 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:24.900 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:25.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 894 16 2502 2486 0.0217222 0 - 0.194117 2024-04-22T02:14:26.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 895 16 2502 2486 0.0216979 0 - 0.194117 2024-04-22T02:14:27.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 896 16 2502 2486 0.0216737 0 - 0.194117 2024-04-22T02:14:28.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 897 16 2502 2486 0.0216495 0 - 0.194117 2024-04-22T02:14:29.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 898 16 2502 2486 0.0216254 0 - 0.194117 2024-04-22T02:14:29.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:30.013 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:30.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 899 16 2502 2486 0.0216014 0 - 0.194117 2024-04-22T02:14:31.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:14:31.335405+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:14:31.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:14:31.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 900 16 2502 2486 0.0215774 0 - 0.194117 2024-04-22T02:14:32.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 901 16 2502 2486 0.0215534 0 - 0.194117 2024-04-22T02:14:33.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 902 16 2502 2486 0.0215295 0 - 0.194117 2024-04-22T02:14:34.333 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 903 16 2502 2486 0.0215057 0 - 0.194117 2024-04-22T02:14:35.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:35.122 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:35.125 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:35.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 904 16 2502 2486 0.0214819 0 - 0.194117 2024-04-22T02:14:36.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 905 16 2502 2486 0.0214582 0 - 0.194117 2024-04-22T02:14:37.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 906 16 2502 2486 0.0214345 0 - 0.194117 2024-04-22T02:14:38.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 907 16 2502 2486 0.0214108 0 - 0.194117 2024-04-22T02:14:39.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 908 16 2502 2486 0.0213873 0 - 0.194117 2024-04-22T02:14:40.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:40.234 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:40.238 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:40.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 909 16 2502 2486 0.0213637 0 - 0.194117 2024-04-22T02:14:41.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 910 16 2502 2486 0.0213403 0 - 0.194117 2024-04-22T02:14:42.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 911 16 2502 2486 0.0213168 0 - 0.194117 2024-04-22T02:14:43.334 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 912 16 2502 2486 0.0212935 0 - 0.194117 2024-04-22T02:14:44.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 913 16 2502 2486 0.0212701 0 - 0.194117 2024-04-22T02:14:45.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:45.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 914 16 2502 2486 0.0212469 0 - 0.194117 2024-04-22T02:14:45.348 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:45.351 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:46.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 915 16 2502 2486 0.0212237 0 - 0.194117 2024-04-22T02:14:47.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 916 16 2502 2486 0.0212005 0 - 0.194117 2024-04-22T02:14:48.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 917 16 2502 2486 0.0211774 0 - 0.194117 2024-04-22T02:14:49.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 918 16 2502 2486 0.0211543 0 - 0.194117 2024-04-22T02:14:50.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 919 16 2502 2486 0.0211313 0 - 0.194117 2024-04-22T02:14:50.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:50.461 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:50.465 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:51.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:14:51.337344+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:14:51.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:14:51.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 920 16 2502 2486 0.0211083 0 - 0.194117 2024-04-22T02:14:52.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 921 16 2502 2486 0.0210854 0 - 0.194117 2024-04-22T02:14:53.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 922 16 2502 2486 0.0210625 0 - 0.194117 2024-04-22T02:14:54.335 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 923 16 2502 2486 0.0210397 0 - 0.194117 2024-04-22T02:14:55.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 924 16 2502 2486 0.0210169 0 - 0.194117 2024-04-22T02:14:55.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:14:55.574 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:14:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:14:55.577 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:14:56.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 925 16 2502 2486 0.0209942 0 - 0.194117 2024-04-22T02:14:57.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 926 16 2502 2486 0.0209715 0 - 0.194117 2024-04-22T02:14:58.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 927 16 2502 2486 0.0209489 0 - 0.194117 2024-04-22T02:14:59.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 928 16 2502 2486 0.0209263 0 - 0.194117 2024-04-22T02:15:00.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 929 16 2502 2486 0.0209038 0 - 0.194117 2024-04-22T02:15:00.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:00.687 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:00.691 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:01.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 930 16 2502 2486 0.0208813 0 - 0.194117 2024-04-22T02:15:02.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 931 16 2502 2486 0.0208589 0 - 0.194117 2024-04-22T02:15:03.336 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 932 16 2502 2486 0.0208365 0 - 0.194117 2024-04-22T02:15:04.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 933 16 2502 2486 0.0208142 0 - 0.194117 2024-04-22T02:15:05.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 934 16 2502 2486 0.0207919 0 - 0.194117 2024-04-22T02:15:05.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:05.801 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:05.804 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:06.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 935 16 2502 2486 0.0207697 0 - 0.194117 2024-04-22T02:15:07.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 936 16 2502 2486 0.0207475 0 - 0.194117 2024-04-22T02:15:08.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 937 16 2502 2486 0.0207253 0 - 0.194117 2024-04-22T02:15:09.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 938 16 2502 2486 0.0207032 0 - 0.194117 2024-04-22T02:15:10.337 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 939 16 2502 2486 0.0206812 0 - 0.194117 2024-04-22T02:15:10.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:10.913 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:10.917 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:11.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:15:11.339652+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:15:11.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:15:11.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 940 16 2502 2486 0.0206592 0 - 0.194117 2024-04-22T02:15:12.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 941 16 2502 2486 0.0206372 0 - 0.194117 2024-04-22T02:15:13.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 942 16 2502 2486 0.0206153 0 - 0.194117 2024-04-22T02:15:14.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 943 16 2502 2486 0.0205935 0 - 0.194117 2024-04-22T02:15:15.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 944 16 2502 2486 0.0205717 0 - 0.194117 2024-04-22T02:15:15.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:16.027 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:16.031 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:16.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 945 16 2502 2486 0.0205499 0 - 0.194117 2024-04-22T02:15:17.338 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 946 16 2502 2486 0.0205282 0 - 0.194117 2024-04-22T02:15:18.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 947 16 2502 2486 0.0205065 0 - 0.194117 2024-04-22T02:15:19.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 948 16 2502 2486 0.0204849 0 - 0.194117 2024-04-22T02:15:20.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 949 16 2502 2486 0.0204633 0 - 0.194117 2024-04-22T02:15:21.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:21.139 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:21.143 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:21.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 950 16 2502 2486 0.0204417 0 - 0.194117 2024-04-22T02:15:22.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 951 16 2502 2486 0.0204202 0 - 0.194117 2024-04-22T02:15:23.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 952 16 2502 2486 0.0203988 0 - 0.194117 2024-04-22T02:15:24.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 953 16 2502 2486 0.0203774 0 - 0.194117 2024-04-22T02:15:25.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 954 16 2502 2486 0.020356 0 - 0.194117 2024-04-22T02:15:26.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:26.251 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:26.254 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:26.339 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 955 16 2502 2486 0.0203347 0 - 0.194117 2024-04-22T02:15:27.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 956 16 2502 2486 0.0203134 0 - 0.194117 2024-04-22T02:15:28.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 957 16 2502 2486 0.0202922 0 - 0.194117 2024-04-22T02:15:29.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 958 16 2502 2486 0.020271 0 - 0.194117 2024-04-22T02:15:30.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 959 16 2502 2486 0.0202499 0 - 0.194117 2024-04-22T02:15:31.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:31.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:15:31.342337+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:15:31.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:15:31.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 960 16 2502 2486 0.0202288 0 - 0.194117 2024-04-22T02:15:31.363 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:31.366 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:32.340 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 961 16 2502 2486 0.0202077 0 - 0.194117 2024-04-22T02:15:33.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 962 16 2502 2486 0.0201867 0 - 0.194117 2024-04-22T02:15:34.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 963 16 2502 2486 0.0201658 0 - 0.194117 2024-04-22T02:15:35.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 964 16 2502 2486 0.0201448 0 - 0.194117 2024-04-22T02:15:36.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 965 16 2502 2486 0.020124 0 - 0.194117 2024-04-22T02:15:36.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:36.475 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:36.478 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:37.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 966 16 2502 2486 0.0201031 0 - 0.194117 2024-04-22T02:15:38.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 967 16 2502 2486 0.0200824 0 - 0.194117 2024-04-22T02:15:39.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 968 16 2502 2486 0.0200616 0 - 0.194117 2024-04-22T02:15:40.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 969 16 2502 2486 0.0200409 0 - 0.194117 2024-04-22T02:15:41.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 970 16 2502 2486 0.0200202 0 - 0.194117 2024-04-22T02:15:41.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:41.586 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:41.590 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:42.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 971 16 2502 2486 0.0199996 0 - 0.194117 2024-04-22T02:15:43.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 972 16 2502 2486 0.0199791 0 - 0.194117 2024-04-22T02:15:44.341 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 973 16 2502 2486 0.0199585 0 - 0.194117 2024-04-22T02:15:45.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 974 16 2502 2486 0.019938 0 - 0.194117 2024-04-22T02:15:46.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 975 16 2502 2486 0.0199176 0 - 0.194117 2024-04-22T02:15:46.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:46.698 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:46.702 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:47.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 976 16 2502 2486 0.0198972 0 - 0.194117 2024-04-22T02:15:48.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 977 16 2502 2486 0.0198768 0 - 0.194117 2024-04-22T02:15:49.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 978 16 2502 2486 0.0198565 0 - 0.194117 2024-04-22T02:15:50.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 979 16 2502 2486 0.0198362 0 - 0.194117 2024-04-22T02:15:51.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:15:51.344302+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:15:51.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:15:51.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 980 16 2502 2486 0.019816 0 - 0.194117 2024-04-22T02:15:51.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:51.811 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:51.814 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:52.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 981 16 2502 2486 0.0197958 0 - 0.194117 2024-04-22T02:15:53.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 982 16 2502 2486 0.0197756 0 - 0.194117 2024-04-22T02:15:54.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 983 16 2502 2486 0.0197555 0 - 0.194117 2024-04-22T02:15:55.342 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 984 16 2502 2486 0.0197354 0 - 0.194117 2024-04-22T02:15:56.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 985 16 2502 2486 0.0197154 0 - 0.194117 2024-04-22T02:15:56.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:15:56.921 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:15:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:15:56.925 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:15:57.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 986 16 2502 2486 0.0196954 0 - 0.194117 2024-04-22T02:15:58.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 987 16 2502 2486 0.0196754 0 - 0.194117 2024-04-22T02:15:59.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 988 16 2502 2486 0.0196555 0 - 0.194117 2024-04-22T02:16:00.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 989 16 2502 2486 0.0196356 0 - 0.194117 2024-04-22T02:16:01.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 990 16 2502 2486 0.0196158 0 - 0.194117 2024-04-22T02:16:01.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:02.038 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:02.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 991 16 2502 2486 0.019596 0 - 0.194117 2024-04-22T02:16:03.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 992 16 2502 2486 0.0195763 0 - 0.194117 2024-04-22T02:16:04.343 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 993 16 2502 2486 0.0195565 0 - 0.194117 2024-04-22T02:16:05.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 994 16 2502 2486 0.0195369 0 - 0.194117 2024-04-22T02:16:06.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 995 16 2502 2486 0.0195172 0 - 0.194117 2024-04-22T02:16:07.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:07.149 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:07.152 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:07.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 996 16 2502 2486 0.0194976 0 - 0.194117 2024-04-22T02:16:08.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 997 16 2502 2486 0.0194781 0 - 0.194117 2024-04-22T02:16:09.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 998 16 2502 2486 0.0194586 0 - 0.194117 2024-04-22T02:16:10.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 999 16 2502 2486 0.0194391 0 - 0.194117 2024-04-22T02:16:11.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:16:11.346511+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:16:11.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:16:11.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1000 16 2502 2486 0.0194196 0 - 0.194117 2024-04-22T02:16:12.153 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:12.260 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:12.264 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:12.344 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1001 16 2502 2486 0.0194002 0 - 0.194117 2024-04-22T02:16:13.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1002 16 2502 2486 0.0193809 0 - 0.194117 2024-04-22T02:16:14.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1003 16 2502 2486 0.0193616 0 - 0.194117 2024-04-22T02:16:15.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1004 16 2502 2486 0.0193423 0 - 0.194117 2024-04-22T02:16:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:16:15.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:16:16.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1005 16 2502 2486 0.019323 0 - 0.194117 2024-04-22T02:16:17.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:17.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1006 16 2502 2486 0.0193038 0 - 0.194117 2024-04-22T02:16:17.373 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:17.376 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:17.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:16:17.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:16:17.726+0000 7f25d599d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:16:18.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1007 16 2502 2486 0.0192847 0 - 0.194117 2024-04-22T02:16:19.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1008 16 2502 2486 0.0192655 0 - 0.194117 2024-04-22T02:16:20.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1009 16 2502 2486 0.0192464 0 - 0.194117 2024-04-22T02:16:21.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1010 16 2502 2486 0.0192274 0 - 0.194117 2024-04-22T02:16:22.345 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1011 16 2502 2486 0.0192084 0 - 0.194117 2024-04-22T02:16:22.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:22.486 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:22.490 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:23.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1012 16 2502 2486 0.0191894 0 - 0.194117 2024-04-22T02:16:24.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1013 16 2502 2486 0.0191704 0 - 0.194117 2024-04-22T02:16:25.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1014 16 2502 2486 0.0191515 0 - 0.194117 2024-04-22T02:16:26.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1015 16 2502 2486 0.0191327 0 - 0.194117 2024-04-22T02:16:27.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1016 16 2502 2486 0.0191138 0 - 0.194117 2024-04-22T02:16:27.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:27.599 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:27.602 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:28.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1017 16 2502 2486 0.019095 0 - 0.194117 2024-04-22T02:16:29.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1018 16 2502 2486 0.0190763 0 - 0.194117 2024-04-22T02:16:30.346 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1019 16 2502 2486 0.0190576 0 - 0.194117 2024-04-22T02:16:31.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:16:31.348806+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:16:31.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:16:31.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1020 16 2502 2486 0.0190389 0 - 0.194117 2024-04-22T02:16:32.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1021 16 2502 2486 0.0190202 0 - 0.194117 2024-04-22T02:16:32.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:32.711 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:32.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:32.715 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:33.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1022 16 2502 2486 0.0190016 0 - 0.194117 2024-04-22T02:16:34.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1023 16 2502 2486 0.018983 0 - 0.194117 2024-04-22T02:16:35.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1024 16 2502 2486 0.0189645 0 - 0.194117 2024-04-22T02:16:36.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1025 16 2502 2486 0.018946 0 - 0.194117 2024-04-22T02:16:37.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1026 16 2502 2486 0.0189275 0 - 0.194117 2024-04-22T02:16:37.716 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:37.825 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:37.829 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:38.347 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1027 16 2502 2486 0.0189091 0 - 0.194117 2024-04-22T02:16:39.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1028 16 2502 2486 0.0188907 0 - 0.194117 2024-04-22T02:16:40.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1029 16 2502 2486 0.0188723 0 - 0.194117 2024-04-22T02:16:41.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1030 16 2502 2486 0.018854 0 - 0.194117 2024-04-22T02:16:42.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1031 16 2502 2486 0.0188357 0 - 0.194117 2024-04-22T02:16:42.830 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:42.938 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:42.942 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:43.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1032 16 2502 2486 0.0188175 0 - 0.194117 2024-04-22T02:16:44.348 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1033 16 2502 2486 0.0187993 0 - 0.194117 2024-04-22T02:16:45.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1034 16 2502 2486 0.0187811 0 - 0.194117 2024-04-22T02:16:46.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1035 16 2502 2486 0.0187629 0 - 0.194117 2024-04-22T02:16:47.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1036 16 2502 2486 0.0187448 0 - 0.194117 2024-04-22T02:16:47.944 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:48.052 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:48.056 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:48.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1037 16 2502 2486 0.0187268 0 - 0.194117 2024-04-22T02:16:49.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1038 16 2502 2486 0.0187087 0 - 0.194117 2024-04-22T02:16:50.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1039 16 2502 2486 0.0186907 0 - 0.194117 2024-04-22T02:16:51.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:16:51.351360+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:16:51.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:16:51.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1040 16 2502 2486 0.0186727 0 - 0.194117 2024-04-22T02:16:52.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1041 16 2502 2486 0.0186548 0 - 0.194117 2024-04-22T02:16:53.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:53.166 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:53.169 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:53.349 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1042 16 2502 2486 0.0186369 0 - 0.194117 2024-04-22T02:16:54.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1043 16 2502 2486 0.018619 0 - 0.194117 2024-04-22T02:16:55.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1044 16 2502 2486 0.0186012 0 - 0.194117 2024-04-22T02:16:56.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1045 16 2502 2486 0.0185834 0 - 0.194117 2024-04-22T02:16:57.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1046 16 2502 2486 0.0185656 0 - 0.194117 2024-04-22T02:16:58.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:16:58.278 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:16:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:16:58.282 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:16:58.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1047 16 2502 2486 0.0185479 0 - 0.194117 2024-04-22T02:16:59.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1048 16 2502 2486 0.0185302 0 - 0.194117 2024-04-22T02:17:00.350 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1049 16 2502 2486 0.0185125 0 - 0.194117 2024-04-22T02:17:01.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1050 16 2502 2486 0.0184949 0 - 0.194117 2024-04-22T02:17:02.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1051 16 2502 2486 0.0184773 0 - 0.194117 2024-04-22T02:17:03.283 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:03.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1052 16 2502 2486 0.0184597 0 - 0.194117 2024-04-22T02:17:03.392 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:03.395 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:04.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1053 16 2502 2486 0.0184422 0 - 0.194117 2024-04-22T02:17:05.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1054 16 2502 2486 0.0184247 0 - 0.194117 2024-04-22T02:17:06.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1055 16 2502 2486 0.0184072 0 - 0.194117 2024-04-22T02:17:07.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1056 16 2502 2486 0.0183898 0 - 0.194117 2024-04-22T02:17:08.351 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1057 16 2502 2486 0.0183724 0 - 0.194117 2024-04-22T02:17:08.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:08.504 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:08.507 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:09.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1058 16 2502 2486 0.018355 0 - 0.194117 2024-04-22T02:17:10.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1059 16 2502 2486 0.0183377 0 - 0.194117 2024-04-22T02:17:11.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:17:11.353830+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:17:11.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:17:11.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1060 16 2502 2486 0.0183204 0 - 0.194117 2024-04-22T02:17:12.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1061 16 2502 2486 0.0183031 0 - 0.194117 2024-04-22T02:17:13.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1062 16 2502 2486 0.0182859 0 - 0.194117 2024-04-22T02:17:13.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:13.616 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:13.619 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:14.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1063 16 2502 2486 0.0182687 0 - 0.194117 2024-04-22T02:17:15.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1064 16 2502 2486 0.0182515 0 - 0.194117 2024-04-22T02:17:16.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1065 16 2502 2486 0.0182344 0 - 0.194117 2024-04-22T02:17:17.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1066 16 2502 2486 0.0182173 0 - 0.194117 2024-04-22T02:17:18.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1067 16 2502 2486 0.0182002 0 - 0.194117 2024-04-22T02:17:18.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:18.729 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:18.732 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:19.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1068 16 2502 2486 0.0181832 0 - 0.194117 2024-04-22T02:17:20.352 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1069 16 2502 2486 0.0181662 0 - 0.194117 2024-04-22T02:17:21.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1070 16 2502 2486 0.0181492 0 - 0.194117 2024-04-22T02:17:22.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1071 16 2502 2486 0.0181322 0 - 0.194117 2024-04-22T02:17:23.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1072 16 2502 2486 0.0181153 0 - 0.194117 2024-04-22T02:17:23.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:23.841 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:23.846 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:24.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1073 16 2502 2486 0.0180985 0 - 0.194117 2024-04-22T02:17:25.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1074 16 2502 2486 0.0180816 0 - 0.194117 2024-04-22T02:17:26.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1075 16 2502 2486 0.0180648 0 - 0.194117 2024-04-22T02:17:27.353 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1076 16 2502 2486 0.018048 0 - 0.194117 2024-04-22T02:17:28.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1077 16 2502 2486 0.0180312 0 - 0.194117 2024-04-22T02:17:28.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:28.958 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:28.960 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:29.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1078 16 2502 2486 0.0180145 0 - 0.194117 2024-04-22T02:17:30.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1079 16 2502 2486 0.0179978 0 - 0.194117 2024-04-22T02:17:31.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:17:31.355962+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:17:31.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:17:31.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1080 16 2502 2486 0.0179811 0 - 0.194117 2024-04-22T02:17:32.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1081 16 2502 2486 0.0179645 0 - 0.194117 2024-04-22T02:17:33.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1082 16 2502 2486 0.0179479 0 - 0.194117 2024-04-22T02:17:33.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:34.068 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:34.071 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:34.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1083 16 2502 2486 0.0179313 0 - 0.194117 2024-04-22T02:17:35.354 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1084 16 2502 2486 0.0179148 0 - 0.194117 2024-04-22T02:17:36.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1085 16 2502 2486 0.0178983 0 - 0.194117 2024-04-22T02:17:37.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1086 16 2502 2486 0.0178818 0 - 0.194117 2024-04-22T02:17:38.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1087 16 2502 2486 0.0178654 0 - 0.194117 2024-04-22T02:17:39.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:39.180 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:39.183 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:39.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1088 16 2502 2486 0.0178489 0 - 0.194117 2024-04-22T02:17:40.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1089 16 2502 2486 0.0178325 0 - 0.194117 2024-04-22T02:17:41.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1090 16 2502 2486 0.0178162 0 - 0.194117 2024-04-22T02:17:42.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1091 16 2502 2486 0.0177999 0 - 0.194117 2024-04-22T02:17:43.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1092 16 2502 2486 0.0177836 0 - 0.194117 2024-04-22T02:17:44.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:44.292 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:44.295 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:44.355 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1093 16 2502 2486 0.0177673 0 - 0.194117 2024-04-22T02:17:45.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1094 16 2502 2486 0.017751 0 - 0.194117 2024-04-22T02:17:46.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1095 16 2502 2486 0.0177348 0 - 0.194117 2024-04-22T02:17:47.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1096 16 2502 2486 0.0177186 0 - 0.194117 2024-04-22T02:17:48.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1097 16 2502 2486 0.0177025 0 - 0.194117 2024-04-22T02:17:49.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:49.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1098 16 2502 2486 0.0176864 0 - 0.194117 2024-04-22T02:17:49.406 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:49.410 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:50.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1099 16 2502 2486 0.0176703 0 - 0.194117 2024-04-22T02:17:51.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:17:51.358429+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:17:51.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:17:51.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1100 16 2502 2486 0.0176542 0 - 0.194117 2024-04-22T02:17:52.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1101 16 2502 2486 0.0176382 0 - 0.194117 2024-04-22T02:17:53.356 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1102 16 2502 2486 0.0176222 0 - 0.194117 2024-04-22T02:17:54.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1103 16 2502 2486 0.0176062 0 - 0.194117 2024-04-22T02:17:54.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:54.518 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:54.522 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:17:55.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1104 16 2502 2486 0.0175903 0 - 0.194117 2024-04-22T02:17:56.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1105 16 2502 2486 0.0175743 0 - 0.194117 2024-04-22T02:17:57.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1106 16 2502 2486 0.0175584 0 - 0.194117 2024-04-22T02:17:58.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1107 16 2502 2486 0.0175426 0 - 0.194117 2024-04-22T02:17:59.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1108 16 2502 2486 0.0175268 0 - 0.194117 2024-04-22T02:17:59.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:17:59.632 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:17:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:17:59.635 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:00.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1109 16 2502 2486 0.0175109 0 - 0.194117 2024-04-22T02:18:01.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1110 16 2502 2486 0.0174952 0 - 0.194117 2024-04-22T02:18:02.357 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1111 16 2502 2486 0.0174794 0 - 0.194117 2024-04-22T02:18:03.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1112 16 2502 2486 0.0174637 0 - 0.194117 2024-04-22T02:18:04.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1113 16 2502 2486 0.017448 0 - 0.194117 2024-04-22T02:18:04.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:04.743 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:04.747 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:05.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1114 16 2502 2486 0.0174324 0 - 0.194117 2024-04-22T02:18:06.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1115 16 2502 2486 0.0174167 0 - 0.194117 2024-04-22T02:18:07.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1116 16 2502 2486 0.0174011 0 - 0.194117 2024-04-22T02:18:08.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1117 16 2502 2486 0.0173855 0 - 0.194117 2024-04-22T02:18:09.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1118 16 2502 2486 0.01737 0 - 0.194117 2024-04-22T02:18:09.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:09.855 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:09.858 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:10.358 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1119 16 2502 2486 0.0173545 0 - 0.194117 2024-04-22T02:18:11.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:18:11.360693+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:18:11.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:18:11.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1120 16 2502 2486 0.017339 0 - 0.194117 2024-04-22T02:18:12.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1121 16 2502 2486 0.0173235 0 - 0.194117 2024-04-22T02:18:13.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1122 16 2502 2486 0.0173081 0 - 0.194117 2024-04-22T02:18:14.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1123 16 2502 2486 0.0172926 0 - 0.194117 2024-04-22T02:18:14.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:14.970 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:15.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1124 16 2502 2486 0.0172773 0 - 0.194117 2024-04-22T02:18:16.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1125 16 2502 2486 0.0172619 0 - 0.194117 2024-04-22T02:18:17.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1126 16 2502 2486 0.0172466 0 - 0.194117 2024-04-22T02:18:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:18:17.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:18:18.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1127 16 2502 2486 0.0172313 0 - 0.194117 2024-04-22T02:18:19.359 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1128 16 2502 2486 0.017216 0 - 0.194117 2024-04-22T02:18:19.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:18:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:18:19.762+0000 7f4eaf6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:18:19.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:20.083 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:20.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1129 16 2502 2486 0.0172007 0 - 0.194117 2024-04-22T02:18:21.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1130 16 2502 2486 0.0171855 0 - 0.194117 2024-04-22T02:18:22.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1131 16 2502 2486 0.0171703 0 - 0.194117 2024-04-22T02:18:23.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1132 16 2502 2486 0.0171552 0 - 0.194117 2024-04-22T02:18:24.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1133 16 2502 2486 0.01714 0 - 0.194117 2024-04-22T02:18:25.084 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:25.192 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:25.196 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-04-22T02:18:25.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1134 16 2502 2486 0.0171249 0 - 0.194117 2024-04-22T02:18:26.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1135 16 2502 2486 0.0171098 0 - 0.194117 2024-04-22T02:18:27.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1136 16 2502 2486 0.0170948 0 - 0.194117 2024-04-22T02:18:28.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1137 16 2502 2486 0.0170797 0 - 0.194117 2024-04-22T02:18:29.360 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1138 16 2502 2486 0.0170647 0 - 0.194117 2024-04-22T02:18:30.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-22T02:18:30.304 INFO:teuthology.orchestra.run.smithi150.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-22T02:18:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-22T02:18:30.309 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-22T02:18:30.309 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1461, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 560, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 3122, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2028, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.3 restart 2024-04-22T02:18:30.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1139 16 2502 2486 0.0170497 0 - 0.194117 2024-04-22T02:18:31.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:18:31.362775+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:18:31.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:18:31.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1140 16 2502 2486 0.0170348 0 - 0.194117 2024-04-22T02:18:32.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1141 16 2502 2486 0.0170198 0 - 0.194117 2024-04-22T02:18:33.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1142 16 2502 2486 0.0170049 0 - 0.194117 2024-04-22T02:18:34.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1143 16 2502 2486 0.0169901 0 - 0.194117 2024-04-22T02:18:35.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1144 16 2502 2486 0.0169752 0 - 0.194117 2024-04-22T02:18:36.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1145 16 2502 2486 0.0169604 0 - 0.194117 2024-04-22T02:18:37.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1146 16 2502 2486 0.0169456 0 - 0.194117 2024-04-22T02:18:38.361 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1147 16 2502 2486 0.0169308 0 - 0.194117 2024-04-22T02:18:39.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1148 16 2502 2486 0.0169161 0 - 0.194117 2024-04-22T02:18:40.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1149 16 2502 2486 0.0169013 0 - 0.194117 2024-04-22T02:18:41.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1150 16 2502 2486 0.0168866 0 - 0.194117 2024-04-22T02:18:42.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1151 16 2502 2486 0.016872 0 - 0.194117 2024-04-22T02:18:43.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1152 16 2502 2486 0.0168573 0 - 0.194117 2024-04-22T02:18:44.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1153 16 2502 2486 0.0168427 0 - 0.194117 2024-04-22T02:18:45.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1154 16 2502 2486 0.0168281 0 - 0.194117 2024-04-22T02:18:46.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1155 16 2502 2486 0.0168135 0 - 0.194117 2024-04-22T02:18:47.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1156 16 2502 2486 0.016799 0 - 0.194117 2024-04-22T02:18:48.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1157 16 2502 2486 0.0167845 0 - 0.194117 2024-04-22T02:18:49.362 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1158 16 2502 2486 0.01677 0 - 0.194117 2024-04-22T02:18:50.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1159 16 2502 2486 0.0167555 0 - 0.194117 2024-04-22T02:18:51.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:18:51.364781+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:18:51.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:18:51.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1160 16 2502 2486 0.0167411 0 - 0.194117 2024-04-22T02:18:52.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1161 16 2502 2486 0.0167267 0 - 0.194117 2024-04-22T02:18:53.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1162 16 2502 2486 0.0167123 0 - 0.194117 2024-04-22T02:18:54.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1163 16 2502 2486 0.0166979 0 - 0.194117 2024-04-22T02:18:55.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1164 16 2502 2486 0.0166835 0 - 0.194117 2024-04-22T02:18:56.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1165 16 2502 2486 0.0166692 0 - 0.194117 2024-04-22T02:18:57.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1166 16 2502 2486 0.0166549 0 - 0.194117 2024-04-22T02:18:58.363 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1167 16 2502 2486 0.0166407 0 - 0.194117 2024-04-22T02:18:59.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1168 16 2502 2486 0.0166264 0 - 0.194117 2024-04-22T02:19:00.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1169 16 2502 2486 0.0166122 0 - 0.194117 2024-04-22T02:19:01.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1170 16 2502 2486 0.016598 0 - 0.194117 2024-04-22T02:19:02.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1171 16 2502 2486 0.0165838 0 - 0.194117 2024-04-22T02:19:03.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1172 16 2502 2486 0.0165697 0 - 0.194117 2024-04-22T02:19:04.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1173 16 2502 2486 0.0165555 0 - 0.194117 2024-04-22T02:19:05.364 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1174 16 2502 2486 0.0165414 0 - 0.194117 2024-04-22T02:19:06.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1175 16 2502 2486 0.0165274 0 - 0.194117 2024-04-22T02:19:07.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1176 16 2502 2486 0.0165133 0 - 0.194117 2024-04-22T02:19:08.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1177 16 2502 2486 0.0164993 0 - 0.194117 2024-04-22T02:19:09.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1178 16 2502 2486 0.0164853 0 - 0.194117 2024-04-22T02:19:10.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1179 16 2502 2486 0.0164713 0 - 0.194117 2024-04-22T02:19:11.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:19:11.367345+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:19:11.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:19:11.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1180 16 2502 2486 0.0164573 0 - 0.194117 2024-04-22T02:19:12.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1181 16 2502 2486 0.0164434 0 - 0.194117 2024-04-22T02:19:13.365 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1182 16 2502 2486 0.0164295 0 - 0.194117 2024-04-22T02:19:14.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1183 16 2502 2486 0.0164156 0 - 0.194117 2024-04-22T02:19:15.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1184 16 2502 2486 0.0164017 0 - 0.194117 2024-04-22T02:19:16.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1185 16 2502 2486 0.0163879 0 - 0.194117 2024-04-22T02:19:17.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1186 16 2502 2486 0.0163741 0 - 0.194117 2024-04-22T02:19:18.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1187 16 2502 2486 0.0163603 0 - 0.194117 2024-04-22T02:19:19.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1188 16 2502 2486 0.0163465 0 - 0.194117 2024-04-22T02:19:20.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1189 16 2502 2486 0.0163328 0 - 0.194117 2024-04-22T02:19:21.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1190 16 2502 2486 0.016319 0 - 0.194117 2024-04-22T02:19:22.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1191 16 2502 2486 0.0163053 0 - 0.194117 2024-04-22T02:19:23.366 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1192 16 2502 2486 0.0162916 0 - 0.194117 2024-04-22T02:19:24.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1193 16 2502 2486 0.016278 0 - 0.194117 2024-04-22T02:19:25.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1194 16 2502 2486 0.0162644 0 - 0.194117 2024-04-22T02:19:26.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1195 16 2502 2486 0.0162507 0 - 0.194117 2024-04-22T02:19:27.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1196 16 2502 2486 0.0162372 0 - 0.194117 2024-04-22T02:19:28.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1197 16 2502 2486 0.0162236 0 - 0.194117 2024-04-22T02:19:29.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1198 16 2502 2486 0.0162101 0 - 0.194117 2024-04-22T02:19:30.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1199 16 2502 2486 0.0161965 0 - 0.194117 2024-04-22T02:19:31.367 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:19:31.369559+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:19:31.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:19:31.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1200 16 2502 2486 0.016183 0 - 0.194117 2024-04-22T02:19:32.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1201 16 2502 2486 0.0161696 0 - 0.194117 2024-04-22T02:19:33.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1202 16 2502 2486 0.0161561 0 - 0.194117 2024-04-22T02:19:34.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1203 16 2502 2486 0.0161427 0 - 0.194117 2024-04-22T02:19:35.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1204 16 2502 2486 0.0161293 0 - 0.194117 2024-04-22T02:19:36.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1205 16 2502 2486 0.0161159 0 - 0.194117 2024-04-22T02:19:37.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1206 16 2502 2486 0.0161025 0 - 0.194117 2024-04-22T02:19:38.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1207 16 2502 2486 0.0160892 0 - 0.194117 2024-04-22T02:19:39.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1208 16 2502 2486 0.0160759 0 - 0.194117 2024-04-22T02:19:40.368 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1209 16 2502 2486 0.0160626 0 - 0.194117 2024-04-22T02:19:41.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1210 16 2502 2486 0.0160493 0 - 0.194117 2024-04-22T02:19:42.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1211 16 2502 2486 0.016036 0 - 0.194117 2024-04-22T02:19:43.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1212 16 2502 2486 0.0160228 0 - 0.194117 2024-04-22T02:19:44.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1213 16 2502 2486 0.0160096 0 - 0.194117 2024-04-22T02:19:45.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1214 16 2502 2486 0.0159964 0 - 0.194117 2024-04-22T02:19:46.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1215 16 2502 2486 0.0159832 0 - 0.194117 2024-04-22T02:19:47.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1216 16 2502 2486 0.0159701 0 - 0.194117 2024-04-22T02:19:48.369 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1217 16 2502 2486 0.015957 0 - 0.194117 2024-04-22T02:19:49.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1218 16 2502 2486 0.0159439 0 - 0.194117 2024-04-22T02:19:50.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1219 16 2502 2486 0.0159308 0 - 0.194117 2024-04-22T02:19:51.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:19:51.371775+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:19:51.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:19:51.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1220 16 2502 2486 0.0159177 0 - 0.194117 2024-04-22T02:19:52.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1221 16 2502 2486 0.0159047 0 - 0.194117 2024-04-22T02:19:53.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1222 16 2502 2486 0.0158917 0 - 0.194117 2024-04-22T02:19:54.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1223 16 2502 2486 0.0158787 0 - 0.194117 2024-04-22T02:19:55.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1224 16 2502 2486 0.0158657 0 - 0.194117 2024-04-22T02:19:56.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1225 16 2502 2486 0.0158528 0 - 0.194117 2024-04-22T02:19:57.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1226 16 2502 2486 0.0158398 0 - 0.194117 2024-04-22T02:19:58.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1227 16 2502 2486 0.0158269 0 - 0.194117 2024-04-22T02:19:59.370 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1228 16 2502 2486 0.015814 0 - 0.194117 2024-04-22T02:20:00.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1229 16 2502 2486 0.0158012 0 - 0.194117 2024-04-22T02:20:01.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1230 16 2502 2486 0.0157883 0 - 0.194117 2024-04-22T02:20:02.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1231 16 2502 2486 0.0157755 0 - 0.194117 2024-04-22T02:20:03.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1232 16 2502 2486 0.0157627 0 - 0.194117 2024-04-22T02:20:04.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1233 16 2502 2486 0.0157499 0 - 0.194117 2024-04-22T02:20:05.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1234 16 2502 2486 0.0157372 0 - 0.194117 2024-04-22T02:20:06.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1235 16 2502 2486 0.0157244 0 - 0.194117 2024-04-22T02:20:07.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1236 16 2502 2486 0.0157117 0 - 0.194117 2024-04-22T02:20:08.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1237 16 2502 2486 0.015699 0 - 0.194117 2024-04-22T02:20:09.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1238 16 2502 2486 0.0156863 0 - 0.194117 2024-04-22T02:20:10.371 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1239 16 2502 2486 0.0156736 0 - 0.194117 2024-04-22T02:20:11.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:20:11.373637+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:20:11.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:20:11.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1240 16 2502 2486 0.015661 0 - 0.194117 2024-04-22T02:20:12.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1241 16 2502 2486 0.0156484 0 - 0.194117 2024-04-22T02:20:13.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1242 16 2502 2486 0.0156358 0 - 0.194117 2024-04-22T02:20:14.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1243 16 2502 2486 0.0156232 0 - 0.194117 2024-04-22T02:20:15.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1244 16 2502 2486 0.0156107 0 - 0.194117 2024-04-22T02:20:16.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1245 16 2502 2486 0.0155981 0 - 0.194117 2024-04-22T02:20:17.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1246 16 2502 2486 0.0155856 0 - 0.194117 2024-04-22T02:20:18.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1247 16 2502 2486 0.0155731 0 - 0.194117 2024-04-22T02:20:19.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1248 16 2502 2486 0.0155606 0 - 0.194117 2024-04-22T02:20:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:20:19.681 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:20:20.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1249 16 2502 2486 0.0155482 0 - 0.194117 2024-04-22T02:20:21.372 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1250 16 2502 2486 0.0155357 0 - 0.194117 2024-04-22T02:20:21.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:20:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:20:21.798+0000 7fe482b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:20:22.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1251 16 2502 2486 0.0155233 0 - 0.194117 2024-04-22T02:20:23.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1252 16 2502 2486 0.0155109 0 - 0.194117 2024-04-22T02:20:24.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1253 16 2502 2486 0.0154985 0 - 0.194117 2024-04-22T02:20:25.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1254 16 2502 2486 0.0154862 0 - 0.194117 2024-04-22T02:20:26.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1255 16 2502 2486 0.0154738 0 - 0.194117 2024-04-22T02:20:27.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1256 16 2502 2486 0.0154615 0 - 0.194117 2024-04-22T02:20:28.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1257 16 2502 2486 0.0154492 0 - 0.194117 2024-04-22T02:20:29.373 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1258 16 2502 2486 0.0154369 0 - 0.194117 2024-04-22T02:20:30.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1259 16 2502 2486 0.0154247 0 - 0.194117 2024-04-22T02:20:31.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:20:31.375733+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:20:31.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:20:31.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1260 16 2502 2486 0.0154124 0 - 0.194117 2024-04-22T02:20:32.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1261 16 2502 2486 0.0154002 0 - 0.194117 2024-04-22T02:20:33.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1262 16 2502 2486 0.015388 0 - 0.194117 2024-04-22T02:20:34.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1263 16 2502 2486 0.0153758 0 - 0.194117 2024-04-22T02:20:35.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1264 16 2502 2486 0.0153637 0 - 0.194117 2024-04-22T02:20:36.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1265 16 2502 2486 0.0153515 0 - 0.194117 2024-04-22T02:20:37.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1266 16 2502 2486 0.0153394 0 - 0.194117 2024-04-22T02:20:38.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1267 16 2502 2486 0.0153273 0 - 0.194117 2024-04-22T02:20:39.374 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1268 16 2502 2486 0.0153152 0 - 0.194117 2024-04-22T02:20:40.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1269 16 2502 2486 0.0153031 0 - 0.194117 2024-04-22T02:20:41.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1270 16 2502 2486 0.0152911 0 - 0.194117 2024-04-22T02:20:42.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1271 16 2502 2486 0.015279 0 - 0.194117 2024-04-22T02:20:43.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1272 16 2502 2486 0.015267 0 - 0.194117 2024-04-22T02:20:44.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1273 16 2502 2486 0.015255 0 - 0.194117 2024-04-22T02:20:45.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1274 16 2502 2486 0.0152431 0 - 0.194117 2024-04-22T02:20:46.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1275 16 2502 2486 0.0152311 0 - 0.194117 2024-04-22T02:20:47.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1276 16 2502 2486 0.0152192 0 - 0.194117 2024-04-22T02:20:48.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1277 16 2502 2486 0.0152073 0 - 0.194117 2024-04-22T02:20:49.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1278 16 2502 2486 0.0151954 0 - 0.194117 2024-04-22T02:20:50.375 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1279 16 2502 2486 0.0151835 0 - 0.194117 2024-04-22T02:20:51.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:20:51.377688+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:20:51.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:20:51.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1280 16 2502 2486 0.0151716 0 - 0.194117 2024-04-22T02:20:52.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1281 16 2502 2486 0.0151598 0 - 0.194117 2024-04-22T02:20:53.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1282 16 2502 2486 0.0151479 0 - 0.194117 2024-04-22T02:20:54.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1283 16 2502 2486 0.0151361 0 - 0.194117 2024-04-22T02:20:55.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1284 16 2502 2486 0.0151243 0 - 0.194117 2024-04-22T02:20:56.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1285 16 2502 2486 0.0151126 0 - 0.194117 2024-04-22T02:20:57.376 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1286 16 2502 2486 0.0151008 0 - 0.194117 2024-04-22T02:20:58.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1287 16 2502 2486 0.0150891 0 - 0.194117 2024-04-22T02:20:59.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1288 16 2502 2486 0.0150774 0 - 0.194117 2024-04-22T02:21:00.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1289 16 2502 2486 0.0150657 0 - 0.194117 2024-04-22T02:21:01.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1290 16 2502 2486 0.015054 0 - 0.194117 2024-04-22T02:21:02.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1291 16 2502 2486 0.0150423 0 - 0.194117 2024-04-22T02:21:03.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1292 16 2502 2486 0.0150307 0 - 0.194117 2024-04-22T02:21:04.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1293 16 2502 2486 0.0150191 0 - 0.194117 2024-04-22T02:21:05.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1294 16 2502 2486 0.0150075 0 - 0.194117 2024-04-22T02:21:06.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1295 16 2502 2486 0.0149959 0 - 0.194117 2024-04-22T02:21:07.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1296 16 2502 2486 0.0149843 0 - 0.194117 2024-04-22T02:21:08.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1297 16 2502 2486 0.0149728 0 - 0.194117 2024-04-22T02:21:09.377 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1298 16 2502 2486 0.0149612 0 - 0.194117 2024-04-22T02:21:10.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1299 16 2502 2486 0.0149497 0 - 0.194117 2024-04-22T02:21:11.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:21:11.379804+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:21:11.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:21:11.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1300 16 2502 2486 0.0149382 0 - 0.194117 2024-04-22T02:21:12.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1301 16 2502 2486 0.0149267 0 - 0.194117 2024-04-22T02:21:13.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1302 16 2502 2486 0.0149153 0 - 0.194117 2024-04-22T02:21:14.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1303 16 2502 2486 0.0149038 0 - 0.194117 2024-04-22T02:21:15.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1304 16 2502 2486 0.0148924 0 - 0.194117 2024-04-22T02:21:16.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1305 16 2502 2486 0.014881 0 - 0.194117 2024-04-22T02:21:17.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1306 16 2502 2486 0.0148696 0 - 0.194117 2024-04-22T02:21:18.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1307 16 2502 2486 0.0148582 0 - 0.194117 2024-04-22T02:21:19.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1308 16 2502 2486 0.0148468 0 - 0.194117 2024-04-22T02:21:20.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1309 16 2502 2486 0.0148355 0 - 0.194117 2024-04-22T02:21:21.378 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1310 16 2502 2486 0.0148242 0 - 0.194117 2024-04-22T02:21:22.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1311 16 2502 2486 0.0148129 0 - 0.194117 2024-04-22T02:21:23.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1312 16 2502 2486 0.0148016 0 - 0.194117 2024-04-22T02:21:24.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1313 16 2502 2486 0.0147903 0 - 0.194117 2024-04-22T02:21:25.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1314 16 2502 2486 0.014779 0 - 0.194117 2024-04-22T02:21:26.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1315 16 2502 2486 0.0147678 0 - 0.194117 2024-04-22T02:21:27.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1316 16 2502 2486 0.0147566 0 - 0.194117 2024-04-22T02:21:28.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1317 16 2502 2486 0.0147454 0 - 0.194117 2024-04-22T02:21:29.379 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1318 16 2502 2486 0.0147342 0 - 0.194117 2024-04-22T02:21:30.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1319 16 2502 2486 0.014723 0 - 0.194117 2024-04-22T02:21:31.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:21:31.381840+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:21:31.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:21:31.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1320 16 2502 2486 0.0147119 0 - 0.194117 2024-04-22T02:21:32.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1321 16 2502 2486 0.0147007 0 - 0.194117 2024-04-22T02:21:33.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1322 16 2502 2486 0.0146896 0 - 0.194117 2024-04-22T02:21:34.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1323 16 2502 2486 0.0146785 0 - 0.194117 2024-04-22T02:21:35.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1324 16 2502 2486 0.0146674 0 - 0.194117 2024-04-22T02:21:36.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1325 16 2502 2486 0.0146564 0 - 0.194117 2024-04-22T02:21:37.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1326 16 2502 2486 0.0146453 0 - 0.194117 2024-04-22T02:21:38.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1327 16 2502 2486 0.0146343 0 - 0.194117 2024-04-22T02:21:39.380 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1328 16 2502 2486 0.0146232 0 - 0.194117 2024-04-22T02:21:40.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1329 16 2502 2486 0.0146122 0 - 0.194117 2024-04-22T02:21:41.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1330 16 2502 2486 0.0146013 0 - 0.194117 2024-04-22T02:21:42.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1331 16 2502 2486 0.0145903 0 - 0.194117 2024-04-22T02:21:43.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1332 16 2502 2486 0.0145793 0 - 0.194117 2024-04-22T02:21:44.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1333 16 2502 2486 0.0145684 0 - 0.194117 2024-04-22T02:21:45.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1334 16 2502 2486 0.0145575 0 - 0.194117 2024-04-22T02:21:46.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1335 16 2502 2486 0.0145466 0 - 0.194117 2024-04-22T02:21:47.381 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1336 16 2502 2486 0.0145357 0 - 0.194117 2024-04-22T02:21:48.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1337 16 2502 2486 0.0145248 0 - 0.194117 2024-04-22T02:21:49.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1338 16 2502 2486 0.014514 0 - 0.194117 2024-04-22T02:21:50.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1339 16 2502 2486 0.0145031 0 - 0.194117 2024-04-22T02:21:51.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:21:51.383902+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:21:51.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:21:51.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1340 16 2502 2486 0.0144923 0 - 0.194117 2024-04-22T02:21:52.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1341 16 2502 2486 0.0144815 0 - 0.194117 2024-04-22T02:21:53.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1342 16 2502 2486 0.0144707 0 - 0.194117 2024-04-22T02:21:54.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1343 16 2502 2486 0.0144599 0 - 0.194117 2024-04-22T02:21:55.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1344 16 2502 2486 0.0144492 0 - 0.194117 2024-04-22T02:21:56.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1345 16 2502 2486 0.0144384 0 - 0.194117 2024-04-22T02:21:57.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1346 16 2502 2486 0.0144277 0 - 0.194117 2024-04-22T02:21:58.382 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1347 16 2502 2486 0.014417 0 - 0.194117 2024-04-22T02:21:59.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1348 16 2502 2486 0.0144063 0 - 0.194117 2024-04-22T02:22:00.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1349 16 2502 2486 0.0143956 0 - 0.194117 2024-04-22T02:22:01.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1350 16 2502 2486 0.0143849 0 - 0.194117 2024-04-22T02:22:02.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1351 16 2502 2486 0.0143743 0 - 0.194117 2024-04-22T02:22:03.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1352 16 2502 2486 0.0143637 0 - 0.194117 2024-04-22T02:22:04.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1353 16 2502 2486 0.014353 0 - 0.194117 2024-04-22T02:22:05.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1354 16 2502 2486 0.0143424 0 - 0.194117 2024-04-22T02:22:06.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1355 16 2502 2486 0.0143319 0 - 0.194117 2024-04-22T02:22:07.383 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1356 16 2502 2486 0.0143213 0 - 0.194117 2024-04-22T02:22:08.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1357 16 2502 2486 0.0143107 0 - 0.194117 2024-04-22T02:22:09.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1358 16 2502 2486 0.0143002 0 - 0.194117 2024-04-22T02:22:10.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1359 16 2502 2486 0.0142897 0 - 0.194117 2024-04-22T02:22:11.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:22:11.386307+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:22:11.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:22:11.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1360 16 2502 2486 0.0142792 0 - 0.194117 2024-04-22T02:22:12.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1361 16 2502 2486 0.0142687 0 - 0.194117 2024-04-22T02:22:13.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1362 16 2502 2486 0.0142582 0 - 0.194117 2024-04-22T02:22:14.384 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1363 16 2502 2486 0.0142477 0 - 0.194117 2024-04-22T02:22:15.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1364 16 2502 2486 0.0142373 0 - 0.194117 2024-04-22T02:22:16.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1365 16 2502 2486 0.0142269 0 - 0.194117 2024-04-22T02:22:17.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1366 16 2502 2486 0.0142164 0 - 0.194117 2024-04-22T02:22:18.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1367 16 2502 2486 0.014206 0 - 0.194117 2024-04-22T02:22:19.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1368 16 2502 2486 0.0141957 0 - 0.194117 2024-04-22T02:22:20.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1369 16 2502 2486 0.0141853 0 - 0.194117 2024-04-22T02:22:21.385 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1370 16 2502 2486 0.0141749 0 - 0.194117 2024-04-22T02:22:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:22:21.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:22:22.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1371 16 2502 2486 0.0141646 0 - 0.194117 2024-04-22T02:22:23.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1372 16 2502 2486 0.0141543 0 - 0.194117 2024-04-22T02:22:23.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:22:23.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:22:23.833+0000 7fb22c2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:22:24.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1373 16 2502 2486 0.014144 0 - 0.194117 2024-04-22T02:22:25.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1374 16 2502 2486 0.0141337 0 - 0.194117 2024-04-22T02:22:26.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1375 16 2502 2486 0.0141234 0 - 0.194117 2024-04-22T02:22:27.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1376 16 2502 2486 0.0141131 0 - 0.194117 2024-04-22T02:22:28.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1377 16 2502 2486 0.0141029 0 - 0.194117 2024-04-22T02:22:29.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1378 16 2502 2486 0.0140926 0 - 0.194117 2024-04-22T02:22:30.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1379 16 2502 2486 0.0140824 0 - 0.194117 2024-04-22T02:22:31.386 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:22:31.388681+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:22:31.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:22:31.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1380 16 2502 2486 0.0140722 0 - 0.194117 2024-04-22T02:22:32.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1381 16 2502 2486 0.014062 0 - 0.194117 2024-04-22T02:22:33.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1382 16 2502 2486 0.0140519 0 - 0.194117 2024-04-22T02:22:34.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1383 16 2502 2486 0.0140417 0 - 0.194117 2024-04-22T02:22:35.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1384 16 2502 2486 0.0140316 0 - 0.194117 2024-04-22T02:22:36.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1385 16 2502 2486 0.0140214 0 - 0.194117 2024-04-22T02:22:37.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1386 16 2502 2486 0.0140113 0 - 0.194117 2024-04-22T02:22:38.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1387 16 2502 2486 0.0140012 0 - 0.194117 2024-04-22T02:22:39.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1388 16 2502 2486 0.0139911 0 - 0.194117 2024-04-22T02:22:40.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1389 16 2502 2486 0.013981 0 - 0.194117 2024-04-22T02:22:41.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1390 16 2502 2486 0.013971 0 - 0.194117 2024-04-22T02:22:42.387 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1391 16 2502 2486 0.0139609 0 - 0.194117 2024-04-22T02:22:43.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1392 16 2502 2486 0.0139509 0 - 0.194117 2024-04-22T02:22:44.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1393 16 2502 2486 0.0139409 0 - 0.194117 2024-04-22T02:22:45.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1394 16 2502 2486 0.0139309 0 - 0.194117 2024-04-22T02:22:46.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1395 16 2502 2486 0.0139209 0 - 0.194117 2024-04-22T02:22:47.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1396 16 2502 2486 0.0139109 0 - 0.194117 2024-04-22T02:22:48.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1397 16 2502 2486 0.013901 0 - 0.194117 2024-04-22T02:22:49.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1398 16 2502 2486 0.013891 0 - 0.194117 2024-04-22T02:22:50.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1399 16 2502 2486 0.0138811 0 - 0.194117 2024-04-22T02:22:51.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:22:51.390546+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:22:51.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:22:51.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1400 16 2502 2486 0.0138712 0 - 0.194117 2024-04-22T02:22:52.388 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1401 16 2502 2486 0.0138613 0 - 0.194117 2024-04-22T02:22:53.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1402 16 2502 2486 0.0138514 0 - 0.194117 2024-04-22T02:22:54.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1403 16 2502 2486 0.0138415 0 - 0.194117 2024-04-22T02:22:55.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1404 16 2502 2486 0.0138317 0 - 0.194117 2024-04-22T02:22:56.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1405 16 2502 2486 0.0138218 0 - 0.194117 2024-04-22T02:22:57.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1406 16 2502 2486 0.013812 0 - 0.194117 2024-04-22T02:22:58.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1407 16 2502 2486 0.0138022 0 - 0.194117 2024-04-22T02:22:59.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1408 16 2502 2486 0.0137924 0 - 0.194117 2024-04-22T02:23:00.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1409 16 2502 2486 0.0137826 0 - 0.194117 2024-04-22T02:23:01.389 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1410 16 2502 2486 0.0137728 0 - 0.194117 2024-04-22T02:23:02.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1411 16 2502 2486 0.0137631 0 - 0.194117 2024-04-22T02:23:03.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1412 16 2502 2486 0.0137533 0 - 0.194117 2024-04-22T02:23:04.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1413 16 2502 2486 0.0137436 0 - 0.194117 2024-04-22T02:23:05.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1414 16 2502 2486 0.0137339 0 - 0.194117 2024-04-22T02:23:06.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1415 16 2502 2486 0.0137241 0 - 0.194117 2024-04-22T02:23:07.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1416 16 2502 2486 0.0137145 0 - 0.194117 2024-04-22T02:23:08.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1417 16 2502 2486 0.0137048 0 - 0.194117 2024-04-22T02:23:09.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1418 16 2502 2486 0.0136951 0 - 0.194117 2024-04-22T02:23:10.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1419 16 2502 2486 0.0136855 0 - 0.194117 2024-04-22T02:23:11.390 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:23:11.392814+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:23:11.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:23:11.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1420 16 2502 2486 0.0136758 0 - 0.194117 2024-04-22T02:23:12.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1421 16 2502 2486 0.0136662 0 - 0.194117 2024-04-22T02:23:13.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1422 16 2502 2486 0.0136566 0 - 0.194117 2024-04-22T02:23:14.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1423 16 2502 2486 0.013647 0 - 0.194117 2024-04-22T02:23:15.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1424 16 2502 2486 0.0136374 0 - 0.194117 2024-04-22T02:23:16.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1425 16 2502 2486 0.0136278 0 - 0.194117 2024-04-22T02:23:17.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1426 16 2502 2486 0.0136183 0 - 0.194117 2024-04-22T02:23:18.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1427 16 2502 2486 0.0136087 0 - 0.194117 2024-04-22T02:23:19.391 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1428 16 2502 2486 0.0135992 0 - 0.194117 2024-04-22T02:23:20.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1429 16 2502 2486 0.0135897 0 - 0.194117 2024-04-22T02:23:21.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1430 16 2502 2486 0.0135802 0 - 0.194117 2024-04-22T02:23:22.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1431 16 2502 2486 0.0135707 0 - 0.194117 2024-04-22T02:23:23.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1432 16 2502 2486 0.0135612 0 - 0.194117 2024-04-22T02:23:24.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1433 16 2502 2486 0.0135518 0 - 0.194117 2024-04-22T02:23:25.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1434 16 2502 2486 0.0135423 0 - 0.194117 2024-04-22T02:23:26.392 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1435 16 2502 2486 0.0135329 0 - 0.194117 2024-04-22T02:23:27.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1436 16 2502 2486 0.0135234 0 - 0.194117 2024-04-22T02:23:28.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1437 16 2502 2486 0.013514 0 - 0.194117 2024-04-22T02:23:29.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1438 16 2502 2486 0.0135046 0 - 0.194117 2024-04-22T02:23:30.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1439 16 2502 2486 0.0134953 0 - 0.194117 2024-04-22T02:23:31.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:23:31.395090+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:23:31.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:23:31.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1440 16 2502 2486 0.0134859 0 - 0.194117 2024-04-22T02:23:32.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1441 16 2502 2486 0.0134765 0 - 0.194117 2024-04-22T02:23:33.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1442 16 2502 2486 0.0134672 0 - 0.194117 2024-04-22T02:23:34.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1443 16 2502 2486 0.0134578 0 - 0.194117 2024-04-22T02:23:35.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1444 16 2502 2486 0.0134485 0 - 0.194117 2024-04-22T02:23:36.393 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1445 16 2502 2486 0.0134392 0 - 0.194117 2024-04-22T02:23:37.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1446 16 2502 2486 0.0134299 0 - 0.194117 2024-04-22T02:23:38.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1447 16 2502 2486 0.0134206 0 - 0.194117 2024-04-22T02:23:39.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1448 16 2502 2486 0.0134114 0 - 0.194117 2024-04-22T02:23:40.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1449 16 2502 2486 0.0134021 0 - 0.194117 2024-04-22T02:23:41.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1450 16 2502 2486 0.0133929 0 - 0.194117 2024-04-22T02:23:42.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1451 16 2502 2486 0.0133836 0 - 0.194117 2024-04-22T02:23:43.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1452 16 2502 2486 0.0133744 0 - 0.194117 2024-04-22T02:23:44.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1453 16 2502 2486 0.0133652 0 - 0.194117 2024-04-22T02:23:45.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1454 16 2502 2486 0.013356 0 - 0.194117 2024-04-22T02:23:46.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1455 16 2502 2486 0.0133469 0 - 0.194117 2024-04-22T02:23:47.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1456 16 2502 2486 0.0133377 0 - 0.194117 2024-04-22T02:23:48.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1457 16 2502 2486 0.0133285 0 - 0.194117 2024-04-22T02:23:49.394 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1458 16 2502 2486 0.0133194 0 - 0.194117 2024-04-22T02:23:50.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1459 16 2502 2486 0.0133103 0 - 0.194117 2024-04-22T02:23:51.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:23:51.396992+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:23:51.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:23:51.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1460 16 2502 2486 0.0133011 0 - 0.194117 2024-04-22T02:23:52.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1461 16 2502 2486 0.013292 0 - 0.194117 2024-04-22T02:23:53.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1462 16 2502 2486 0.013283 0 - 0.194117 2024-04-22T02:23:54.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1463 16 2502 2486 0.0132739 0 - 0.194117 2024-04-22T02:23:55.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1464 16 2502 2486 0.0132648 0 - 0.194117 2024-04-22T02:23:56.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1465 16 2502 2486 0.0132558 0 - 0.194117 2024-04-22T02:23:57.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1466 16 2502 2486 0.0132467 0 - 0.194117 2024-04-22T02:23:58.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1467 16 2502 2486 0.0132377 0 - 0.194117 2024-04-22T02:23:59.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1468 16 2502 2486 0.0132287 0 - 0.194117 2024-04-22T02:24:00.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1469 16 2502 2486 0.0132197 0 - 0.194117 2024-04-22T02:24:01.395 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1470 16 2502 2486 0.0132107 0 - 0.194117 2024-04-22T02:24:02.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1471 16 2502 2486 0.0132017 0 - 0.194117 2024-04-22T02:24:03.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1472 16 2502 2486 0.0131927 0 - 0.194117 2024-04-22T02:24:04.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1473 16 2502 2486 0.0131838 0 - 0.194117 2024-04-22T02:24:05.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1474 16 2502 2486 0.0131748 0 - 0.194117 2024-04-22T02:24:06.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1475 16 2502 2486 0.0131659 0 - 0.194117 2024-04-22T02:24:07.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1476 16 2502 2486 0.013157 0 - 0.194117 2024-04-22T02:24:08.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1477 16 2502 2486 0.0131481 0 - 0.194117 2024-04-22T02:24:09.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1478 16 2502 2486 0.0131392 0 - 0.194117 2024-04-22T02:24:10.396 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1479 16 2502 2486 0.0131303 0 - 0.194117 2024-04-22T02:24:11.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:24:11.398831+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:24:11.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:24:11.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1480 16 2502 2486 0.0131214 0 - 0.194117 2024-04-22T02:24:12.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1481 16 2502 2486 0.0131125 0 - 0.194117 2024-04-22T02:24:13.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1482 16 2502 2486 0.0131037 0 - 0.194117 2024-04-22T02:24:14.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1483 16 2502 2486 0.0130949 0 - 0.194117 2024-04-22T02:24:15.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1484 16 2502 2486 0.013086 0 - 0.194117 2024-04-22T02:24:16.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1485 16 2502 2486 0.0130772 0 - 0.194117 2024-04-22T02:24:17.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1486 16 2502 2486 0.0130684 0 - 0.194117 2024-04-22T02:24:18.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1487 16 2502 2486 0.0130596 0 - 0.194117 2024-04-22T02:24:19.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1488 16 2502 2486 0.0130509 0 - 0.194117 2024-04-22T02:24:20.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1489 16 2502 2486 0.0130421 0 - 0.194117 2024-04-22T02:24:21.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1490 16 2502 2486 0.0130333 0 - 0.194117 2024-04-22T02:24:22.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1491 16 2502 2486 0.0130246 0 - 0.194117 2024-04-22T02:24:23.397 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1492 16 2502 2486 0.0130159 0 - 0.194117 2024-04-22T02:24:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:24:23.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:24:24.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1493 16 2502 2486 0.0130072 0 - 0.194117 2024-04-22T02:24:25.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1494 16 2502 2486 0.0129985 0 - 0.194117 2024-04-22T02:24:25.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:24:25.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:24:25.870+0000 7f66a146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:24:26.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1495 16 2502 2486 0.0129898 0 - 0.194117 2024-04-22T02:24:27.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1496 16 2502 2486 0.0129811 0 - 0.194117 2024-04-22T02:24:28.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1497 16 2502 2486 0.0129724 0 - 0.194117 2024-04-22T02:24:29.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1498 16 2502 2486 0.0129637 0 - 0.194117 2024-04-22T02:24:30.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1499 16 2502 2486 0.0129551 0 - 0.194117 2024-04-22T02:24:31.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:24:31.400662+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:24:31.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:24:31.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1500 16 2502 2486 0.0129465 0 - 0.194117 2024-04-22T02:24:32.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1501 16 2502 2486 0.0129378 0 - 0.194117 2024-04-22T02:24:33.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1502 16 2502 2486 0.0129292 0 - 0.194117 2024-04-22T02:24:34.398 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1503 16 2502 2486 0.0129206 0 - 0.194117 2024-04-22T02:24:35.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1504 16 2502 2486 0.012912 0 - 0.194117 2024-04-22T02:24:36.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1505 16 2502 2486 0.0129034 0 - 0.194117 2024-04-22T02:24:37.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1506 16 2502 2486 0.0128949 0 - 0.194117 2024-04-22T02:24:38.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1507 16 2502 2486 0.0128863 0 - 0.194117 2024-04-22T02:24:39.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1508 16 2502 2486 0.0128778 0 - 0.194117 2024-04-22T02:24:40.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1509 16 2502 2486 0.0128692 0 - 0.194117 2024-04-22T02:24:41.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1510 16 2502 2486 0.0128607 0 - 0.194117 2024-04-22T02:24:42.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1511 16 2502 2486 0.0128522 0 - 0.194117 2024-04-22T02:24:43.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1512 16 2502 2486 0.0128437 0 - 0.194117 2024-04-22T02:24:44.399 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1513 16 2502 2486 0.0128352 0 - 0.194117 2024-04-22T02:24:45.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1514 16 2502 2486 0.0128267 0 - 0.194117 2024-04-22T02:24:46.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1515 16 2502 2486 0.0128183 0 - 0.194117 2024-04-22T02:24:47.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1516 16 2502 2486 0.0128098 0 - 0.194117 2024-04-22T02:24:48.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1517 16 2502 2486 0.0128014 0 - 0.194117 2024-04-22T02:24:49.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1518 16 2502 2486 0.0127929 0 - 0.194117 2024-04-22T02:24:50.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1519 16 2502 2486 0.0127845 0 - 0.194117 2024-04-22T02:24:51.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:24:51.402388+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:24:51.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:24:51.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1520 16 2502 2486 0.0127761 0 - 0.194117 2024-04-22T02:24:52.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1521 16 2502 2486 0.0127677 0 - 0.194117 2024-04-22T02:24:53.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1522 16 2502 2486 0.0127593 0 - 0.194117 2024-04-22T02:24:54.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1523 16 2502 2486 0.0127509 0 - 0.194117 2024-04-22T02:24:55.400 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1524 16 2502 2486 0.0127426 0 - 0.194117 2024-04-22T02:24:56.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1525 16 2502 2486 0.0127342 0 - 0.194117 2024-04-22T02:24:57.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1526 16 2502 2486 0.0127259 0 - 0.194117 2024-04-22T02:24:58.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1527 16 2502 2486 0.0127175 0 - 0.194117 2024-04-22T02:24:59.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1528 16 2502 2486 0.0127092 0 - 0.194117 2024-04-22T02:25:00.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1529 16 2502 2486 0.0127009 0 - 0.194117 2024-04-22T02:25:01.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1530 16 2502 2486 0.0126926 0 - 0.194117 2024-04-22T02:25:02.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1531 16 2502 2486 0.0126843 0 - 0.194117 2024-04-22T02:25:03.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1532 16 2502 2486 0.012676 0 - 0.194117 2024-04-22T02:25:04.401 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1533 16 2502 2486 0.0126678 0 - 0.194117 2024-04-22T02:25:05.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1534 16 2502 2486 0.0126595 0 - 0.194117 2024-04-22T02:25:06.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1535 16 2502 2486 0.0126513 0 - 0.194117 2024-04-22T02:25:07.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1536 16 2502 2486 0.012643 0 - 0.194117 2024-04-22T02:25:08.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1537 16 2502 2486 0.0126348 0 - 0.194117 2024-04-22T02:25:09.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1538 16 2502 2486 0.0126266 0 - 0.194117 2024-04-22T02:25:10.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1539 16 2502 2486 0.0126184 0 - 0.194117 2024-04-22T02:25:11.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:25:11.404378+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:25:11.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:25:11.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1540 16 2502 2486 0.0126102 0 - 0.194117 2024-04-22T02:25:12.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1541 16 2502 2486 0.012602 0 - 0.194117 2024-04-22T02:25:13.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1542 16 2502 2486 0.0125938 0 - 0.194117 2024-04-22T02:25:14.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1543 16 2502 2486 0.0125857 0 - 0.194117 2024-04-22T02:25:15.402 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1544 16 2502 2486 0.0125775 0 - 0.194117 2024-04-22T02:25:16.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1545 16 2502 2486 0.0125694 0 - 0.194117 2024-04-22T02:25:17.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1546 16 2502 2486 0.0125613 0 - 0.194117 2024-04-22T02:25:18.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1547 16 2502 2486 0.0125531 0 - 0.194117 2024-04-22T02:25:19.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1548 16 2502 2486 0.012545 0 - 0.194117 2024-04-22T02:25:20.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1549 16 2502 2486 0.0125369 0 - 0.194117 2024-04-22T02:25:21.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1550 16 2502 2486 0.0125288 0 - 0.194117 2024-04-22T02:25:22.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1551 16 2502 2486 0.0125208 0 - 0.194117 2024-04-22T02:25:23.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1552 16 2502 2486 0.0125127 0 - 0.194117 2024-04-22T02:25:24.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1553 16 2502 2486 0.0125046 0 - 0.194117 2024-04-22T02:25:25.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1554 16 2502 2486 0.0124966 0 - 0.194117 2024-04-22T02:25:26.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1555 16 2502 2486 0.0124886 0 - 0.194117 2024-04-22T02:25:27.403 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1556 16 2502 2486 0.0124805 0 - 0.194117 2024-04-22T02:25:28.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1557 16 2502 2486 0.0124725 0 - 0.194117 2024-04-22T02:25:29.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1558 16 2502 2486 0.0124645 0 - 0.194117 2024-04-22T02:25:30.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1559 16 2502 2486 0.0124565 0 - 0.194117 2024-04-22T02:25:31.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:25:31.406448+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:25:31.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:25:31.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1560 16 2502 2486 0.0124485 0 - 0.194117 2024-04-22T02:25:32.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1561 16 2502 2486 0.0124405 0 - 0.194117 2024-04-22T02:25:33.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1562 16 2502 2486 0.0124326 0 - 0.194117 2024-04-22T02:25:34.404 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1563 16 2502 2486 0.0124246 0 - 0.194117 2024-04-22T02:25:35.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1564 16 2502 2486 0.0124167 0 - 0.194117 2024-04-22T02:25:36.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1565 16 2502 2486 0.0124088 0 - 0.194117 2024-04-22T02:25:37.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1566 16 2502 2486 0.0124008 0 - 0.194117 2024-04-22T02:25:38.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1567 16 2502 2486 0.0123929 0 - 0.194117 2024-04-22T02:25:39.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1568 16 2502 2486 0.012385 0 - 0.194117 2024-04-22T02:25:40.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1569 16 2502 2486 0.0123771 0 - 0.194117 2024-04-22T02:25:41.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1570 16 2502 2486 0.0123692 0 - 0.194117 2024-04-22T02:25:42.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1571 16 2502 2486 0.0123614 0 - 0.194117 2024-04-22T02:25:43.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1572 16 2502 2486 0.0123535 0 - 0.194117 2024-04-22T02:25:44.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1573 16 2502 2486 0.0123456 0 - 0.194117 2024-04-22T02:25:45.405 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1574 16 2502 2486 0.0123378 0 - 0.194117 2024-04-22T02:25:46.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1575 16 2502 2486 0.01233 0 - 0.194117 2024-04-22T02:25:47.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1576 16 2502 2486 0.0123221 0 - 0.194117 2024-04-22T02:25:48.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1577 16 2502 2486 0.0123143 0 - 0.194117 2024-04-22T02:25:49.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1578 16 2502 2486 0.0123065 0 - 0.194117 2024-04-22T02:25:50.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1579 16 2502 2486 0.0122987 0 - 0.194117 2024-04-22T02:25:51.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:25:51.408531+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:25:51.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:25:51.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1580 16 2502 2486 0.0122909 0 - 0.194117 2024-04-22T02:25:52.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1581 16 2502 2486 0.0122832 0 - 0.194117 2024-04-22T02:25:53.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1582 16 2502 2486 0.0122754 0 - 0.194117 2024-04-22T02:25:54.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1583 16 2502 2486 0.0122677 0 - 0.194117 2024-04-22T02:25:55.406 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1584 16 2502 2486 0.0122599 0 - 0.194117 2024-04-22T02:25:56.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1585 16 2502 2486 0.0122522 0 - 0.194117 2024-04-22T02:25:57.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1586 16 2502 2486 0.0122445 0 - 0.194117 2024-04-22T02:25:58.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1587 16 2502 2486 0.0122367 0 - 0.194117 2024-04-22T02:25:59.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1588 16 2502 2486 0.012229 0 - 0.194117 2024-04-22T02:26:00.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1589 16 2502 2486 0.0122213 0 - 0.194117 2024-04-22T02:26:01.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1590 16 2502 2486 0.0122136 0 - 0.194117 2024-04-22T02:26:02.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1591 16 2502 2486 0.012206 0 - 0.194117 2024-04-22T02:26:03.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1592 16 2502 2486 0.0121983 0 - 0.194117 2024-04-22T02:26:04.407 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1593 16 2502 2486 0.0121906 0 - 0.194117 2024-04-22T02:26:05.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1594 16 2502 2486 0.012183 0 - 0.194117 2024-04-22T02:26:06.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1595 16 2502 2486 0.0121754 0 - 0.194117 2024-04-22T02:26:07.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1596 16 2502 2486 0.0121677 0 - 0.194117 2024-04-22T02:26:08.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1597 16 2502 2486 0.0121601 0 - 0.194117 2024-04-22T02:26:09.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1598 16 2502 2486 0.0121525 0 - 0.194117 2024-04-22T02:26:10.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1599 16 2502 2486 0.0121449 0 - 0.194117 2024-04-22T02:26:11.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:26:11.410471+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:26:11.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:26:11.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1600 16 2502 2486 0.0121373 0 - 0.194117 2024-04-22T02:26:12.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1601 16 2502 2486 0.0121297 0 - 0.194117 2024-04-22T02:26:13.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1602 16 2502 2486 0.0121222 0 - 0.194117 2024-04-22T02:26:14.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1603 16 2502 2486 0.0121146 0 - 0.194117 2024-04-22T02:26:15.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1604 16 2502 2486 0.012107 0 - 0.194117 2024-04-22T02:26:16.408 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1605 16 2502 2486 0.0120995 0 - 0.194117 2024-04-22T02:26:17.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1606 16 2502 2486 0.012092 0 - 0.194117 2024-04-22T02:26:18.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1607 16 2502 2486 0.0120844 0 - 0.194117 2024-04-22T02:26:19.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1608 16 2502 2486 0.0120769 0 - 0.194117 2024-04-22T02:26:20.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1609 16 2502 2486 0.0120694 0 - 0.194117 2024-04-22T02:26:21.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1610 16 2502 2486 0.0120619 0 - 0.194117 2024-04-22T02:26:22.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1611 16 2502 2486 0.0120544 0 - 0.194117 2024-04-22T02:26:23.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1612 16 2502 2486 0.012047 0 - 0.194117 2024-04-22T02:26:24.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1613 16 2502 2486 0.0120395 0 - 0.194117 2024-04-22T02:26:25.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1614 16 2502 2486 0.012032 0 - 0.194117 2024-04-22T02:26:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:26:25.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:26:26.409 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1615 16 2502 2486 0.0120246 0 - 0.194117 2024-04-22T02:26:27.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1616 16 2502 2486 0.0120171 0 - 0.194117 2024-04-22T02:26:27.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:26:27.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:26:27.906+0000 7f2b9e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:26:28.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1617 16 2502 2486 0.0120097 0 - 0.194117 2024-04-22T02:26:29.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1618 16 2502 2486 0.0120023 0 - 0.194117 2024-04-22T02:26:30.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1619 16 2502 2486 0.0119949 0 - 0.194117 2024-04-22T02:26:31.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:26:31.412456+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:26:31.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:26:31.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1620 16 2502 2486 0.0119875 0 - 0.194117 2024-04-22T02:26:32.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1621 16 2502 2486 0.0119801 0 - 0.194117 2024-04-22T02:26:33.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1622 16 2502 2486 0.0119727 0 - 0.194117 2024-04-22T02:26:34.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1623 16 2502 2486 0.0119653 0 - 0.194117 2024-04-22T02:26:35.410 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1624 16 2502 2486 0.0119579 0 - 0.194117 2024-04-22T02:26:36.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1625 16 2502 2486 0.0119506 0 - 0.194117 2024-04-22T02:26:37.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1626 16 2502 2486 0.0119432 0 - 0.194117 2024-04-22T02:26:38.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1627 16 2502 2486 0.0119359 0 - 0.194117 2024-04-22T02:26:39.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1628 16 2502 2486 0.0119286 0 - 0.194117 2024-04-22T02:26:40.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1629 16 2502 2486 0.0119212 0 - 0.194117 2024-04-22T02:26:41.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1630 16 2502 2486 0.0119139 0 - 0.194117 2024-04-22T02:26:42.411 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1631 16 2502 2486 0.0119066 0 - 0.194117 2024-04-22T02:26:43.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1632 16 2502 2486 0.0118993 0 - 0.194117 2024-04-22T02:26:44.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1633 16 2502 2486 0.011892 0 - 0.194117 2024-04-22T02:26:45.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1634 16 2502 2486 0.0118848 0 - 0.194117 2024-04-22T02:26:46.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1635 16 2502 2486 0.0118775 0 - 0.194117 2024-04-22T02:26:47.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1636 16 2502 2486 0.0118702 0 - 0.194117 2024-04-22T02:26:48.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1637 16 2502 2486 0.011863 0 - 0.194117 2024-04-22T02:26:49.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1638 16 2502 2486 0.0118557 0 - 0.194117 2024-04-22T02:26:50.412 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1639 16 2502 2486 0.0118485 0 - 0.194117 2024-04-22T02:26:51.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:26:51.415039+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:26:51.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:26:51.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1640 16 2502 2486 0.0118413 0 - 0.194117 2024-04-22T02:26:52.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1641 16 2502 2486 0.0118341 0 - 0.194117 2024-04-22T02:26:53.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1642 16 2502 2486 0.0118269 0 - 0.194117 2024-04-22T02:26:54.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1643 16 2502 2486 0.0118197 0 - 0.194117 2024-04-22T02:26:55.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1644 16 2502 2486 0.0118125 0 - 0.194117 2024-04-22T02:26:56.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1645 16 2502 2486 0.0118053 0 - 0.194117 2024-04-22T02:26:57.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1646 16 2502 2486 0.0117981 0 - 0.194117 2024-04-22T02:26:58.413 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1647 16 2502 2486 0.011791 0 - 0.194117 2024-04-22T02:26:59.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1648 16 2502 2486 0.0117838 0 - 0.194117 2024-04-22T02:27:00.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1649 16 2502 2486 0.0117767 0 - 0.194117 2024-04-22T02:27:01.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1650 16 2502 2486 0.0117695 0 - 0.194117 2024-04-22T02:27:02.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1651 16 2502 2486 0.0117624 0 - 0.194117 2024-04-22T02:27:03.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1652 16 2502 2486 0.0117553 0 - 0.194117 2024-04-22T02:27:04.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1653 16 2502 2486 0.0117482 0 - 0.194117 2024-04-22T02:27:05.414 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1654 16 2502 2486 0.0117411 0 - 0.194117 2024-04-22T02:27:06.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1655 16 2502 2486 0.011734 0 - 0.194117 2024-04-22T02:27:07.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1656 16 2502 2486 0.0117269 0 - 0.194117 2024-04-22T02:27:08.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1657 16 2502 2486 0.0117198 0 - 0.194117 2024-04-22T02:27:09.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1658 16 2502 2486 0.0117127 0 - 0.194117 2024-04-22T02:27:10.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1659 16 2502 2486 0.0117057 0 - 0.194117 2024-04-22T02:27:11.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:27:11.417565+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:27:11.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:27:11.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1660 16 2502 2486 0.0116986 0 - 0.194117 2024-04-22T02:27:12.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1661 16 2502 2486 0.0116916 0 - 0.194117 2024-04-22T02:27:13.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1662 16 2502 2486 0.0116845 0 - 0.194117 2024-04-22T02:27:14.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1663 16 2502 2486 0.0116775 0 - 0.194117 2024-04-22T02:27:15.415 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1664 16 2502 2486 0.0116705 0 - 0.194117 2024-04-22T02:27:16.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1665 16 2502 2486 0.0116635 0 - 0.194117 2024-04-22T02:27:17.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1666 16 2502 2486 0.0116565 0 - 0.194117 2024-04-22T02:27:18.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1667 16 2502 2486 0.0116495 0 - 0.194117 2024-04-22T02:27:19.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1668 16 2502 2486 0.0116425 0 - 0.194117 2024-04-22T02:27:20.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1669 16 2502 2486 0.0116355 0 - 0.194117 2024-04-22T02:27:21.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1670 16 2502 2486 0.0116286 0 - 0.194117 2024-04-22T02:27:22.416 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1671 16 2502 2486 0.0116216 0 - 0.194117 2024-04-22T02:27:23.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1672 16 2502 2486 0.0116147 0 - 0.194117 2024-04-22T02:27:24.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1673 16 2502 2486 0.0116077 0 - 0.194117 2024-04-22T02:27:25.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1674 16 2502 2486 0.0116008 0 - 0.194117 2024-04-22T02:27:26.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1675 16 2502 2486 0.0115938 0 - 0.194117 2024-04-22T02:27:27.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1676 16 2502 2486 0.0115869 0 - 0.194117 2024-04-22T02:27:28.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1677 16 2502 2486 0.01158 0 - 0.194117 2024-04-22T02:27:29.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1678 16 2502 2486 0.0115731 0 - 0.194117 2024-04-22T02:27:30.417 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1679 16 2502 2486 0.0115662 0 - 0.194117 2024-04-22T02:27:31.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:27:31.420036+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:27:31.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:27:31.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1680 16 2502 2486 0.0115593 0 - 0.194117 2024-04-22T02:27:32.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1681 16 2502 2486 0.0115525 0 - 0.194117 2024-04-22T02:27:33.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1682 16 2502 2486 0.0115456 0 - 0.194117 2024-04-22T02:27:34.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1683 16 2502 2486 0.0115387 0 - 0.194117 2024-04-22T02:27:35.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1684 16 2502 2486 0.0115319 0 - 0.194117 2024-04-22T02:27:36.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1685 16 2502 2486 0.011525 0 - 0.194117 2024-04-22T02:27:37.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1686 16 2502 2486 0.0115182 0 - 0.194117 2024-04-22T02:27:38.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1687 16 2502 2486 0.0115114 0 - 0.194117 2024-04-22T02:27:39.418 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1688 16 2502 2486 0.0115046 0 - 0.194117 2024-04-22T02:27:40.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1689 16 2502 2486 0.0114977 0 - 0.194117 2024-04-22T02:27:41.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1690 16 2502 2486 0.0114909 0 - 0.194117 2024-04-22T02:27:42.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1691 16 2502 2486 0.0114842 0 - 0.194117 2024-04-22T02:27:43.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1692 16 2502 2486 0.0114774 0 - 0.194117 2024-04-22T02:27:44.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1693 16 2502 2486 0.0114706 0 - 0.194117 2024-04-22T02:27:45.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1694 16 2502 2486 0.0114638 0 - 0.194117 2024-04-22T02:27:46.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1695 16 2502 2486 0.011457 0 - 0.194117 2024-04-22T02:27:47.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1696 16 2502 2486 0.0114503 0 - 0.194117 2024-04-22T02:27:48.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1697 16 2502 2486 0.0114435 0 - 0.194117 2024-04-22T02:27:49.419 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1698 16 2502 2486 0.0114368 0 - 0.194117 2024-04-22T02:27:50.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1699 16 2502 2486 0.0114301 0 - 0.194117 2024-04-22T02:27:51.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:27:51.422243+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:27:51.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:27:51.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1700 16 2502 2486 0.0114234 0 - 0.194117 2024-04-22T02:27:52.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1701 16 2502 2486 0.0114166 0 - 0.194117 2024-04-22T02:27:53.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1702 16 2502 2486 0.0114099 0 - 0.194117 2024-04-22T02:27:54.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1703 16 2502 2486 0.0114032 0 - 0.194117 2024-04-22T02:27:55.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1704 16 2502 2486 0.0113965 0 - 0.194117 2024-04-22T02:27:56.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1705 16 2502 2486 0.0113899 0 - 0.194117 2024-04-22T02:27:57.420 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1706 16 2502 2486 0.0113832 0 - 0.194117 2024-04-22T02:27:58.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1707 16 2502 2486 0.0113765 0 - 0.194117 2024-04-22T02:27:59.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1708 16 2502 2486 0.0113698 0 - 0.194117 2024-04-22T02:28:00.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1709 16 2502 2486 0.0113632 0 - 0.194117 2024-04-22T02:28:01.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1710 16 2502 2486 0.0113565 0 - 0.194117 2024-04-22T02:28:02.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1711 16 2502 2486 0.0113499 0 - 0.194117 2024-04-22T02:28:03.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1712 16 2502 2486 0.0113433 0 - 0.194117 2024-04-22T02:28:04.421 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1713 16 2502 2486 0.0113367 0 - 0.194117 2024-04-22T02:28:05.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1714 16 2502 2486 0.01133 0 - 0.194117 2024-04-22T02:28:06.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1715 16 2502 2486 0.0113234 0 - 0.194117 2024-04-22T02:28:07.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1716 16 2502 2486 0.0113168 0 - 0.194117 2024-04-22T02:28:08.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1717 16 2502 2486 0.0113102 0 - 0.194117 2024-04-22T02:28:09.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1718 16 2502 2486 0.0113037 0 - 0.194117 2024-04-22T02:28:10.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1719 16 2502 2486 0.0112971 0 - 0.194117 2024-04-22T02:28:11.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:28:11.424731+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:28:11.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:28:11.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1720 16 2502 2486 0.0112905 0 - 0.194117 2024-04-22T02:28:12.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1721 16 2502 2486 0.011284 0 - 0.194117 2024-04-22T02:28:13.422 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1722 16 2502 2486 0.0112774 0 - 0.194117 2024-04-22T02:28:14.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1723 16 2502 2486 0.0112709 0 - 0.194117 2024-04-22T02:28:15.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1724 16 2502 2486 0.0112643 0 - 0.194117 2024-04-22T02:28:16.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1725 16 2502 2486 0.0112578 0 - 0.194117 2024-04-22T02:28:17.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1726 16 2502 2486 0.0112513 0 - 0.194117 2024-04-22T02:28:18.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1727 16 2502 2486 0.0112448 0 - 0.194117 2024-04-22T02:28:19.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1728 16 2502 2486 0.0112382 0 - 0.194117 2024-04-22T02:28:20.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1729 16 2502 2486 0.0112317 0 - 0.194117 2024-04-22T02:28:21.423 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1730 16 2502 2486 0.0112253 0 - 0.194117 2024-04-22T02:28:22.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1731 16 2502 2486 0.0112188 0 - 0.194117 2024-04-22T02:28:23.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1732 16 2502 2486 0.0112123 0 - 0.194117 2024-04-22T02:28:24.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1733 16 2502 2486 0.0112058 0 - 0.194117 2024-04-22T02:28:25.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1734 16 2502 2486 0.0111994 0 - 0.194117 2024-04-22T02:28:26.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1735 16 2502 2486 0.0111929 0 - 0.194117 2024-04-22T02:28:27.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1736 16 2502 2486 0.0111865 0 - 0.194117 2024-04-22T02:28:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:28:27.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:28:28.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1737 16 2502 2486 0.01118 0 - 0.194117 2024-04-22T02:28:29.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1738 16 2502 2486 0.0111736 0 - 0.194117 2024-04-22T02:28:29.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:28:29.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:28:29.943+0000 7f845e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:28:30.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1739 16 2502 2486 0.0111672 0 - 0.194117 2024-04-22T02:28:31.424 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:28:31.426983+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:28:31.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:28:31.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1740 16 2502 2486 0.0111607 0 - 0.194117 2024-04-22T02:28:32.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1741 16 2502 2486 0.0111543 0 - 0.194117 2024-04-22T02:28:33.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1742 16 2502 2486 0.0111479 0 - 0.194117 2024-04-22T02:28:34.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1743 16 2502 2486 0.0111415 0 - 0.194117 2024-04-22T02:28:35.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1744 16 2502 2486 0.0111351 0 - 0.194117 2024-04-22T02:28:36.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1745 16 2502 2486 0.0111288 0 - 0.194117 2024-04-22T02:28:37.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1746 16 2502 2486 0.0111224 0 - 0.194117 2024-04-22T02:28:38.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1747 16 2502 2486 0.011116 0 - 0.194117 2024-04-22T02:28:39.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1748 16 2502 2486 0.0111097 0 - 0.194117 2024-04-22T02:28:40.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1749 16 2502 2486 0.0111033 0 - 0.194117 2024-04-22T02:28:41.425 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1750 16 2502 2486 0.011097 0 - 0.194117 2024-04-22T02:28:42.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1751 16 2502 2486 0.0110906 0 - 0.194117 2024-04-22T02:28:43.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1752 16 2502 2486 0.0110843 0 - 0.194117 2024-04-22T02:28:44.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1753 16 2502 2486 0.011078 0 - 0.194117 2024-04-22T02:28:45.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1754 16 2502 2486 0.0110717 0 - 0.194117 2024-04-22T02:28:46.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1755 16 2502 2486 0.0110654 0 - 0.194117 2024-04-22T02:28:47.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1756 16 2502 2486 0.0110591 0 - 0.194117 2024-04-22T02:28:48.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1757 16 2502 2486 0.0110528 0 - 0.194117 2024-04-22T02:28:49.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1758 16 2502 2486 0.0110465 0 - 0.194117 2024-04-22T02:28:50.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1759 16 2502 2486 0.0110402 0 - 0.194117 2024-04-22T02:28:51.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:28:51.428999+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:28:51.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:28:51.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1760 16 2502 2486 0.0110339 0 - 0.194117 2024-04-22T02:28:52.426 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1761 16 2502 2486 0.0110277 0 - 0.194117 2024-04-22T02:28:53.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1762 16 2502 2486 0.0110214 0 - 0.194117 2024-04-22T02:28:54.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1763 16 2502 2486 0.0110151 0 - 0.194117 2024-04-22T02:28:55.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1764 16 2502 2486 0.0110089 0 - 0.194117 2024-04-22T02:28:56.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1765 16 2502 2486 0.0110027 0 - 0.194117 2024-04-22T02:28:57.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1766 16 2502 2486 0.0109964 0 - 0.194117 2024-04-22T02:28:58.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1767 16 2502 2486 0.0109902 0 - 0.194117 2024-04-22T02:28:59.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1768 16 2502 2486 0.010984 0 - 0.194117 2024-04-22T02:29:00.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1769 16 2502 2486 0.0109778 0 - 0.194117 2024-04-22T02:29:01.427 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1770 16 2502 2486 0.0109716 0 - 0.194117 2024-04-22T02:29:02.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1771 16 2502 2486 0.0109654 0 - 0.194117 2024-04-22T02:29:03.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1772 16 2502 2486 0.0109592 0 - 0.194117 2024-04-22T02:29:04.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1773 16 2502 2486 0.010953 0 - 0.194117 2024-04-22T02:29:05.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1774 16 2502 2486 0.0109468 0 - 0.194117 2024-04-22T02:29:06.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1775 16 2502 2486 0.0109407 0 - 0.194117 2024-04-22T02:29:07.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1776 16 2502 2486 0.0109345 0 - 0.194117 2024-04-22T02:29:08.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1777 16 2502 2486 0.0109284 0 - 0.194117 2024-04-22T02:29:09.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1778 16 2502 2486 0.0109222 0 - 0.194117 2024-04-22T02:29:10.428 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1779 16 2502 2486 0.0109161 0 - 0.194117 2024-04-22T02:29:11.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:29:11.431078+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:29:11.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:29:11.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1780 16 2502 2486 0.0109099 0 - 0.194117 2024-04-22T02:29:12.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1781 16 2502 2486 0.0109038 0 - 0.194117 2024-04-22T02:29:13.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1782 16 2502 2486 0.0108977 0 - 0.194117 2024-04-22T02:29:14.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1783 16 2502 2486 0.0108916 0 - 0.194117 2024-04-22T02:29:15.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1784 16 2502 2486 0.0108855 0 - 0.194117 2024-04-22T02:29:16.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1785 16 2502 2486 0.0108794 0 - 0.194117 2024-04-22T02:29:17.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1786 16 2502 2486 0.0108733 0 - 0.194117 2024-04-22T02:29:18.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1787 16 2502 2486 0.0108672 0 - 0.194117 2024-04-22T02:29:19.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1788 16 2502 2486 0.0108611 0 - 0.194117 2024-04-22T02:29:20.429 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1789 16 2502 2486 0.0108551 0 - 0.194117 2024-04-22T02:29:21.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1790 16 2502 2486 0.010849 0 - 0.194117 2024-04-22T02:29:22.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1791 16 2502 2486 0.0108429 0 - 0.194117 2024-04-22T02:29:23.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1792 16 2502 2486 0.0108369 0 - 0.194117 2024-04-22T02:29:24.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1793 16 2502 2486 0.0108308 0 - 0.194117 2024-04-22T02:29:25.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1794 16 2502 2486 0.0108248 0 - 0.194117 2024-04-22T02:29:26.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1795 16 2502 2486 0.0108188 0 - 0.194117 2024-04-22T02:29:27.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1796 16 2502 2486 0.0108128 0 - 0.194117 2024-04-22T02:29:28.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1797 16 2502 2486 0.0108067 0 - 0.194117 2024-04-22T02:29:29.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1798 16 2502 2486 0.0108007 0 - 0.194117 2024-04-22T02:29:30.430 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1799 16 2502 2486 0.0107947 0 - 0.194117 2024-04-22T02:29:31.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:29:31.433158+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:29:31.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:29:31.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1800 16 2502 2486 0.0107887 0 - 0.194117 2024-04-22T02:29:32.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1801 16 2502 2486 0.0107827 0 - 0.194117 2024-04-22T02:29:33.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1802 16 2502 2486 0.0107767 0 - 0.194117 2024-04-22T02:29:34.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1803 16 2502 2486 0.0107708 0 - 0.194117 2024-04-22T02:29:35.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1804 16 2502 2486 0.0107648 0 - 0.194117 2024-04-22T02:29:36.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1805 16 2502 2486 0.0107588 0 - 0.194117 2024-04-22T02:29:37.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1806 16 2502 2486 0.0107529 0 - 0.194117 2024-04-22T02:29:38.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1807 16 2502 2486 0.0107469 0 - 0.194117 2024-04-22T02:29:39.431 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1808 16 2502 2486 0.010741 0 - 0.194117 2024-04-22T02:29:40.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1809 16 2502 2486 0.010735 0 - 0.194117 2024-04-22T02:29:41.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1810 16 2502 2486 0.0107291 0 - 0.194117 2024-04-22T02:29:42.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1811 16 2502 2486 0.0107232 0 - 0.194117 2024-04-22T02:29:43.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1812 16 2502 2486 0.0107173 0 - 0.194117 2024-04-22T02:29:44.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1813 16 2502 2486 0.0107114 0 - 0.194117 2024-04-22T02:29:45.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1814 16 2502 2486 0.0107055 0 - 0.194117 2024-04-22T02:29:46.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1815 16 2502 2486 0.0106996 0 - 0.194117 2024-04-22T02:29:47.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1816 16 2502 2486 0.0106937 0 - 0.194117 2024-04-22T02:29:48.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1817 16 2502 2486 0.0106878 0 - 0.194117 2024-04-22T02:29:49.432 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1818 16 2502 2486 0.0106819 0 - 0.194117 2024-04-22T02:29:50.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1819 16 2502 2486 0.010676 0 - 0.194117 2024-04-22T02:29:51.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:29:51.435278+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:29:51.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:29:51.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1820 16 2502 2486 0.0106702 0 - 0.194117 2024-04-22T02:29:52.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1821 16 2502 2486 0.0106643 0 - 0.194117 2024-04-22T02:29:53.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1822 16 2502 2486 0.0106585 0 - 0.194117 2024-04-22T02:29:54.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1823 16 2502 2486 0.0106526 0 - 0.194117 2024-04-22T02:29:55.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1824 16 2502 2486 0.0106468 0 - 0.194117 2024-04-22T02:29:56.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1825 16 2502 2486 0.0106409 0 - 0.194117 2024-04-22T02:29:57.433 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1826 16 2502 2486 0.0106351 0 - 0.194117 2024-04-22T02:29:58.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1827 16 2502 2486 0.0106293 0 - 0.194117 2024-04-22T02:29:59.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1828 16 2502 2486 0.0106235 0 - 0.194117 2024-04-22T02:30:00.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1829 16 2502 2486 0.0106177 0 - 0.194117 2024-04-22T02:30:01.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1830 16 2502 2486 0.0106119 0 - 0.194117 2024-04-22T02:30:02.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1831 16 2502 2486 0.0106061 0 - 0.194117 2024-04-22T02:30:03.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1832 16 2502 2486 0.0106003 0 - 0.194117 2024-04-22T02:30:04.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1833 16 2502 2486 0.0105945 0 - 0.194117 2024-04-22T02:30:05.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1834 16 2502 2486 0.0105887 0 - 0.194117 2024-04-22T02:30:06.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1835 16 2502 2486 0.0105829 0 - 0.194117 2024-04-22T02:30:07.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1836 16 2502 2486 0.0105772 0 - 0.194117 2024-04-22T02:30:08.434 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1837 16 2502 2486 0.0105714 0 - 0.194117 2024-04-22T02:30:09.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1838 16 2502 2486 0.0105657 0 - 0.194117 2024-04-22T02:30:10.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1839 16 2502 2486 0.0105599 0 - 0.194117 2024-04-22T02:30:11.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:30:11.437481+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:30:11.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:30:11.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1840 16 2502 2486 0.0105542 0 - 0.194117 2024-04-22T02:30:12.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1841 16 2502 2486 0.0105485 0 - 0.194117 2024-04-22T02:30:13.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1842 16 2502 2486 0.0105427 0 - 0.194117 2024-04-22T02:30:14.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1843 16 2502 2486 0.010537 0 - 0.194117 2024-04-22T02:30:15.435 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1844 16 2502 2486 0.0105313 0 - 0.194117 2024-04-22T02:30:16.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1845 16 2502 2486 0.0105256 0 - 0.194117 2024-04-22T02:30:17.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1846 16 2502 2486 0.0105199 0 - 0.194117 2024-04-22T02:30:18.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1847 16 2502 2486 0.0105142 0 - 0.194117 2024-04-22T02:30:19.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1848 16 2502 2486 0.0105085 0 - 0.194117 2024-04-22T02:30:20.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1849 16 2502 2486 0.0105028 0 - 0.194117 2024-04-22T02:30:21.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1850 16 2502 2486 0.0104971 0 - 0.194117 2024-04-22T02:30:22.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1851 16 2502 2486 0.0104915 0 - 0.194117 2024-04-22T02:30:23.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1852 16 2502 2486 0.0104858 0 - 0.194117 2024-04-22T02:30:24.436 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1853 16 2502 2486 0.0104801 0 - 0.194117 2024-04-22T02:30:25.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1854 16 2502 2486 0.0104745 0 - 0.194117 2024-04-22T02:30:26.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1855 16 2502 2486 0.0104688 0 - 0.194117 2024-04-22T02:30:27.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1856 16 2502 2486 0.0104632 0 - 0.194117 2024-04-22T02:30:28.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1857 16 2502 2486 0.0104576 0 - 0.194117 2024-04-22T02:30:29.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1858 16 2502 2486 0.0104519 0 - 0.194117 2024-04-22T02:30:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:30:29.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:30:30.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1859 16 2502 2486 0.0104463 0 - 0.194117 2024-04-22T02:30:31.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:30:31.439678+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:30:31.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:30:31.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1860 16 2502 2486 0.0104407 0 - 0.194117 2024-04-22T02:30:31.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:30:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:30:31.978+0000 7f9ba702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:30:32.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1861 16 2502 2486 0.0104351 0 - 0.194117 2024-04-22T02:30:33.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1862 16 2502 2486 0.0104295 0 - 0.194117 2024-04-22T02:30:34.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1863 16 2502 2486 0.0104239 0 - 0.194117 2024-04-22T02:30:35.437 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1864 16 2502 2486 0.0104183 0 - 0.194117 2024-04-22T02:30:36.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1865 16 2502 2486 0.0104127 0 - 0.194117 2024-04-22T02:30:37.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1866 16 2502 2486 0.0104071 0 - 0.194117 2024-04-22T02:30:38.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1867 16 2502 2486 0.0104016 0 - 0.194117 2024-04-22T02:30:39.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1868 16 2502 2486 0.010396 0 - 0.194117 2024-04-22T02:30:40.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1869 16 2502 2486 0.0103904 0 - 0.194117 2024-04-22T02:30:41.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1870 16 2502 2486 0.0103849 0 - 0.194117 2024-04-22T02:30:42.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1871 16 2502 2486 0.0103793 0 - 0.194117 2024-04-22T02:30:43.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1872 16 2502 2486 0.0103738 0 - 0.194117 2024-04-22T02:30:44.438 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1873 16 2502 2486 0.0103682 0 - 0.194117 2024-04-22T02:30:45.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1874 16 2502 2486 0.0103627 0 - 0.194117 2024-04-22T02:30:46.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1875 16 2502 2486 0.0103572 0 - 0.194117 2024-04-22T02:30:47.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1876 16 2502 2486 0.0103517 0 - 0.194117 2024-04-22T02:30:48.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1877 16 2502 2486 0.0103461 0 - 0.194117 2024-04-22T02:30:49.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1878 16 2502 2486 0.0103406 0 - 0.194117 2024-04-22T02:30:50.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1879 16 2502 2486 0.0103351 0 - 0.194117 2024-04-22T02:30:51.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:30:51.441818+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:30:51.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:30:51.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1880 16 2502 2486 0.0103296 0 - 0.194117 2024-04-22T02:30:52.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1881 16 2502 2486 0.0103241 0 - 0.194117 2024-04-22T02:30:53.439 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1882 16 2502 2486 0.0103187 0 - 0.194117 2024-04-22T02:30:54.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1883 16 2502 2486 0.0103132 0 - 0.194117 2024-04-22T02:30:55.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1884 16 2502 2486 0.0103077 0 - 0.194117 2024-04-22T02:30:56.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1885 16 2502 2486 0.0103022 0 - 0.194117 2024-04-22T02:30:57.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1886 16 2502 2486 0.0102968 0 - 0.194117 2024-04-22T02:30:58.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1887 16 2502 2486 0.0102913 0 - 0.194117 2024-04-22T02:30:59.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1888 16 2502 2486 0.0102859 0 - 0.194117 2024-04-22T02:31:00.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1889 16 2502 2486 0.0102804 0 - 0.194117 2024-04-22T02:31:01.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1890 16 2502 2486 0.010275 0 - 0.194117 2024-04-22T02:31:02.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1891 16 2502 2486 0.0102695 0 - 0.194117 2024-04-22T02:31:03.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1892 16 2502 2486 0.0102641 0 - 0.194117 2024-04-22T02:31:04.440 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1893 16 2502 2486 0.0102587 0 - 0.194117 2024-04-22T02:31:05.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1894 16 2502 2486 0.0102533 0 - 0.194117 2024-04-22T02:31:06.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1895 16 2502 2486 0.0102479 0 - 0.194117 2024-04-22T02:31:07.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1896 16 2502 2486 0.0102425 0 - 0.194117 2024-04-22T02:31:08.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1897 16 2502 2486 0.0102371 0 - 0.194117 2024-04-22T02:31:09.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1898 16 2502 2486 0.0102317 0 - 0.194117 2024-04-22T02:31:10.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1899 16 2502 2486 0.0102263 0 - 0.194117 2024-04-22T02:31:11.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:31:11.443734+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:31:11.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:31:11.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1900 16 2502 2486 0.0102209 0 - 0.194117 2024-04-22T02:31:12.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1901 16 2502 2486 0.0102155 0 - 0.194117 2024-04-22T02:31:13.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1902 16 2502 2486 0.0102102 0 - 0.194117 2024-04-22T02:31:14.441 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1903 16 2502 2486 0.0102048 0 - 0.194117 2024-04-22T02:31:15.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1904 16 2502 2486 0.0101994 0 - 0.194117 2024-04-22T02:31:16.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1905 16 2502 2486 0.0101941 0 - 0.194117 2024-04-22T02:31:17.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1906 16 2502 2486 0.0101887 0 - 0.194117 2024-04-22T02:31:18.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1907 16 2502 2486 0.0101834 0 - 0.194117 2024-04-22T02:31:19.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1908 16 2502 2486 0.010178 0 - 0.194117 2024-04-22T02:31:20.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1909 16 2502 2486 0.0101727 0 - 0.194117 2024-04-22T02:31:21.442 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1910 16 2502 2486 0.0101674 0 - 0.194117 2024-04-22T02:31:22.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1911 16 2502 2486 0.0101621 0 - 0.194117 2024-04-22T02:31:23.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1912 16 2502 2486 0.0101567 0 - 0.194117 2024-04-22T02:31:24.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1913 16 2502 2486 0.0101514 0 - 0.194117 2024-04-22T02:31:25.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1914 16 2502 2486 0.0101461 0 - 0.194117 2024-04-22T02:31:26.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1915 16 2502 2486 0.0101408 0 - 0.194117 2024-04-22T02:31:27.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1916 16 2502 2486 0.0101355 0 - 0.194117 2024-04-22T02:31:28.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1917 16 2502 2486 0.0101303 0 - 0.194117 2024-04-22T02:31:29.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1918 16 2502 2486 0.010125 0 - 0.194117 2024-04-22T02:31:30.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1919 16 2502 2486 0.0101197 0 - 0.194117 2024-04-22T02:31:31.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:31:31.446056+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:31:31.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:31:31.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1920 16 2502 2486 0.0101144 0 - 0.194117 2024-04-22T02:31:32.443 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1921 16 2502 2486 0.0101092 0 - 0.194117 2024-04-22T02:31:33.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1922 16 2502 2486 0.0101039 0 - 0.194117 2024-04-22T02:31:34.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1923 16 2502 2486 0.0100987 0 - 0.194117 2024-04-22T02:31:35.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1924 16 2502 2486 0.0100934 0 - 0.194117 2024-04-22T02:31:36.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1925 16 2502 2486 0.0100882 0 - 0.194117 2024-04-22T02:31:37.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1926 16 2502 2486 0.0100829 0 - 0.194117 2024-04-22T02:31:38.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1927 16 2502 2486 0.0100777 0 - 0.194117 2024-04-22T02:31:39.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1928 16 2502 2486 0.0100725 0 - 0.194117 2024-04-22T02:31:40.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1929 16 2502 2486 0.0100672 0 - 0.194117 2024-04-22T02:31:41.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1930 16 2502 2486 0.010062 0 - 0.194117 2024-04-22T02:31:42.444 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1931 16 2502 2486 0.0100568 0 - 0.194117 2024-04-22T02:31:43.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1932 16 2502 2486 0.0100516 0 - 0.194117 2024-04-22T02:31:44.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1933 16 2502 2486 0.0100464 0 - 0.194117 2024-04-22T02:31:45.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1934 16 2502 2486 0.0100412 0 - 0.194117 2024-04-22T02:31:46.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1935 16 2502 2486 0.010036 0 - 0.194117 2024-04-22T02:31:47.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1936 16 2502 2486 0.0100308 0 - 0.194117 2024-04-22T02:31:48.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1937 16 2502 2486 0.0100257 0 - 0.194117 2024-04-22T02:31:49.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1938 16 2502 2486 0.0100205 0 - 0.194117 2024-04-22T02:31:50.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1939 16 2502 2486 0.0100153 0 - 0.194117 2024-04-22T02:31:51.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:31:51.447828+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:31:51.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:31:51.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1940 16 2502 2486 0.0100102 0 - 0.194117 2024-04-22T02:31:52.445 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1941 16 2502 2486 0.010005 0 - 0.194117 2024-04-22T02:31:53.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1942 16 2502 2486 0.00999985 0 - 0.194117 2024-04-22T02:31:54.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1943 16 2502 2486 0.0099947 0 - 0.194117 2024-04-22T02:31:55.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1944 16 2502 2486 0.00998956 0 - 0.194117 2024-04-22T02:31:56.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1945 16 2502 2486 0.00998443 0 - 0.194117 2024-04-22T02:31:57.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1946 16 2502 2486 0.0099793 0 - 0.194117 2024-04-22T02:31:58.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1947 16 2502 2486 0.00997417 0 - 0.194117 2024-04-22T02:31:59.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1948 16 2502 2486 0.00996905 0 - 0.194117 2024-04-22T02:32:00.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1949 16 2502 2486 0.00996394 0 - 0.194117 2024-04-22T02:32:01.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1950 16 2502 2486 0.00995883 0 - 0.194117 2024-04-22T02:32:02.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1951 16 2502 2486 0.00995372 0 - 0.194117 2024-04-22T02:32:03.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1952 16 2502 2486 0.00994862 0 - 0.194117 2024-04-22T02:32:04.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1953 16 2502 2486 0.00994353 0 - 0.194117 2024-04-22T02:32:05.446 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1954 16 2502 2486 0.00993844 0 - 0.194117 2024-04-22T02:32:06.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1955 16 2502 2486 0.00993336 0 - 0.194117 2024-04-22T02:32:07.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1956 16 2502 2486 0.00992828 0 - 0.194117 2024-04-22T02:32:08.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1957 16 2502 2486 0.00992321 0 - 0.194117 2024-04-22T02:32:09.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1958 16 2502 2486 0.00991814 0 - 0.194117 2024-04-22T02:32:10.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1959 16 2502 2486 0.00991308 0 - 0.194117 2024-04-22T02:32:11.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:32:11.449740+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:32:11.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:32:11.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1960 16 2502 2486 0.00990802 0 - 0.194117 2024-04-22T02:32:12.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1961 16 2502 2486 0.00990296 0 - 0.194117 2024-04-22T02:32:13.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1962 16 2502 2486 0.00989792 0 - 0.194117 2024-04-22T02:32:14.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1963 16 2502 2486 0.00989288 0 - 0.194117 2024-04-22T02:32:15.447 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1964 16 2502 2486 0.00988784 0 - 0.194117 2024-04-22T02:32:16.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1965 16 2502 2486 0.00988281 0 - 0.194117 2024-04-22T02:32:17.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1966 16 2502 2486 0.00987778 0 - 0.194117 2024-04-22T02:32:18.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1967 16 2502 2486 0.00987276 0 - 0.194117 2024-04-22T02:32:19.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1968 16 2502 2486 0.00986774 0 - 0.194117 2024-04-22T02:32:20.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1969 16 2502 2486 0.00986273 0 - 0.194117 2024-04-22T02:32:21.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1970 16 2502 2486 0.00985772 0 - 0.194117 2024-04-22T02:32:22.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1971 16 2502 2486 0.00985272 0 - 0.194117 2024-04-22T02:32:23.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1972 16 2502 2486 0.00984773 0 - 0.194117 2024-04-22T02:32:24.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1973 16 2502 2486 0.00984274 0 - 0.194117 2024-04-22T02:32:25.448 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1974 16 2502 2486 0.00983775 0 - 0.194117 2024-04-22T02:32:26.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1975 16 2502 2486 0.00983277 0 - 0.194117 2024-04-22T02:32:27.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1976 16 2502 2486 0.00982779 0 - 0.194117 2024-04-22T02:32:28.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1977 16 2502 2486 0.00982282 0 - 0.194117 2024-04-22T02:32:29.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1978 16 2502 2486 0.00981785 0 - 0.194117 2024-04-22T02:32:30.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1979 16 2502 2486 0.00981289 0 - 0.194117 2024-04-22T02:32:31.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:32:31.451671+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:32:31.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:32:31.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1980 16 2502 2486 0.00980794 0 - 0.194117 2024-04-22T02:32:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:32:31.896 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:32:32.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1981 16 2502 2486 0.00980299 0 - 0.194117 2024-04-22T02:32:33.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1982 16 2502 2486 0.00979804 0 - 0.194117 2024-04-22T02:32:33.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:32:34.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:32:34.013+0000 7fea5fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:32:34.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1983 16 2502 2486 0.0097931 0 - 0.194117 2024-04-22T02:32:35.449 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1984 16 2502 2486 0.00978816 0 - 0.194117 2024-04-22T02:32:36.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1985 16 2502 2486 0.00978323 0 - 0.194117 2024-04-22T02:32:37.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1986 16 2502 2486 0.00977831 0 - 0.194117 2024-04-22T02:32:38.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1987 16 2502 2486 0.00977339 0 - 0.194117 2024-04-22T02:32:39.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1988 16 2502 2486 0.00976847 0 - 0.194117 2024-04-22T02:32:40.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1989 16 2502 2486 0.00976356 0 - 0.194117 2024-04-22T02:32:41.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1990 16 2502 2486 0.00975865 0 - 0.194117 2024-04-22T02:32:42.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1991 16 2502 2486 0.00975375 0 - 0.194117 2024-04-22T02:32:43.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1992 16 2502 2486 0.00974885 0 - 0.194117 2024-04-22T02:32:44.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1993 16 2502 2486 0.00974396 0 - 0.194117 2024-04-22T02:32:45.450 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1994 16 2502 2486 0.00973908 0 - 0.194117 2024-04-22T02:32:46.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1995 16 2502 2486 0.00973419 0 - 0.194117 2024-04-22T02:32:47.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1996 16 2502 2486 0.00972932 0 - 0.194117 2024-04-22T02:32:48.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1997 16 2502 2486 0.00972445 0 - 0.194117 2024-04-22T02:32:49.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1998 16 2502 2486 0.00971958 0 - 0.194117 2024-04-22T02:32:50.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1999 16 2502 2486 0.00971472 0 - 0.194117 2024-04-22T02:32:51.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:32:51.453673+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:32:51.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:32:51.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2000 16 2502 2486 0.00970986 0 - 0.194117 2024-04-22T02:32:52.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2001 16 2502 2486 0.00970501 0 - 0.194117 2024-04-22T02:32:53.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2002 16 2502 2486 0.00970016 0 - 0.194117 2024-04-22T02:32:54.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2003 16 2502 2486 0.00969532 0 - 0.194117 2024-04-22T02:32:55.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2004 16 2502 2486 0.00969048 0 - 0.194117 2024-04-22T02:32:56.451 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2005 16 2502 2486 0.00968565 0 - 0.194117 2024-04-22T02:32:57.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2006 16 2502 2486 0.00968082 0 - 0.194117 2024-04-22T02:32:58.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2007 16 2502 2486 0.00967599 0 - 0.194117 2024-04-22T02:32:59.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2008 16 2502 2486 0.00967118 0 - 0.194117 2024-04-22T02:33:00.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2009 16 2502 2486 0.00966636 0 - 0.194117 2024-04-22T02:33:01.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2010 16 2502 2486 0.00966155 0 - 0.194117 2024-04-22T02:33:02.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2011 16 2502 2486 0.00965675 0 - 0.194117 2024-04-22T02:33:03.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2012 16 2502 2486 0.00965195 0 - 0.194117 2024-04-22T02:33:04.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2013 16 2502 2486 0.00964715 0 - 0.194117 2024-04-22T02:33:05.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2014 16 2502 2486 0.00964236 0 - 0.194117 2024-04-22T02:33:06.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2015 16 2502 2486 0.00963758 0 - 0.194117 2024-04-22T02:33:07.452 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2016 16 2502 2486 0.0096328 0 - 0.194117 2024-04-22T02:33:08.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2017 16 2502 2486 0.00962802 0 - 0.194117 2024-04-22T02:33:09.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2018 16 2502 2486 0.00962325 0 - 0.194117 2024-04-22T02:33:10.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2019 16 2502 2486 0.00961849 0 - 0.194117 2024-04-22T02:33:11.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:33:11.455528+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:33:11.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:33:11.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2020 16 2502 2486 0.00961372 0 - 0.194117 2024-04-22T02:33:12.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2021 16 2502 2486 0.00960897 0 - 0.194117 2024-04-22T02:33:13.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2022 16 2502 2486 0.00960421 0 - 0.194117 2024-04-22T02:33:14.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2023 16 2502 2486 0.00959947 0 - 0.194117 2024-04-22T02:33:15.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2024 16 2502 2486 0.00959472 0 - 0.194117 2024-04-22T02:33:16.453 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2025 16 2502 2486 0.00958999 0 - 0.194117 2024-04-22T02:33:17.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2026 16 2502 2486 0.00958525 0 - 0.194117 2024-04-22T02:33:18.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2027 16 2502 2486 0.00958052 0 - 0.194117 2024-04-22T02:33:19.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2028 16 2502 2486 0.0095758 0 - 0.194117 2024-04-22T02:33:20.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2029 16 2502 2486 0.00957108 0 - 0.194117 2024-04-22T02:33:21.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2030 16 2502 2486 0.00956637 0 - 0.194117 2024-04-22T02:33:22.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2031 16 2502 2486 0.00956166 0 - 0.194117 2024-04-22T02:33:23.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2032 16 2502 2486 0.00955695 0 - 0.194117 2024-04-22T02:33:24.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2033 16 2502 2486 0.00955225 0 - 0.194117 2024-04-22T02:33:25.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2034 16 2502 2486 0.00954755 0 - 0.194117 2024-04-22T02:33:26.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2035 16 2502 2486 0.00954286 0 - 0.194117 2024-04-22T02:33:27.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2036 16 2502 2486 0.00953818 0 - 0.194117 2024-04-22T02:33:28.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2037 16 2502 2486 0.00953349 0 - 0.194117 2024-04-22T02:33:29.454 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2038 16 2502 2486 0.00952881 0 - 0.194117 2024-04-22T02:33:30.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2039 16 2502 2486 0.00952414 0 - 0.194117 2024-04-22T02:33:31.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:33:31.457479+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:33:31.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:33:31.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2040 16 2502 2486 0.00951947 0 - 0.194117 2024-04-22T02:33:32.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2041 16 2502 2486 0.00951481 0 - 0.194117 2024-04-22T02:33:33.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2042 16 2502 2486 0.00951015 0 - 0.194117 2024-04-22T02:33:34.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2043 16 2502 2486 0.0095055 0 - 0.194117 2024-04-22T02:33:35.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2044 16 2502 2486 0.00950084 0 - 0.194117 2024-04-22T02:33:36.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2045 16 2502 2486 0.0094962 0 - 0.194117 2024-04-22T02:33:37.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2046 16 2502 2486 0.00949156 0 - 0.194117 2024-04-22T02:33:38.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2047 16 2502 2486 0.00948692 0 - 0.194117 2024-04-22T02:33:39.455 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2048 16 2502 2486 0.00948229 0 - 0.194117 2024-04-22T02:33:40.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2049 16 2502 2486 0.00947766 0 - 0.194117 2024-04-22T02:33:41.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2050 16 2502 2486 0.00947304 0 - 0.194117 2024-04-22T02:33:42.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2051 16 2502 2486 0.00946842 0 - 0.194117 2024-04-22T02:33:43.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2052 16 2502 2486 0.0094638 0 - 0.194117 2024-04-22T02:33:44.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2053 16 2502 2486 0.00945919 0 - 0.194117 2024-04-22T02:33:45.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2054 16 2502 2486 0.00945459 0 - 0.194117 2024-04-22T02:33:46.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2055 16 2502 2486 0.00944999 0 - 0.194117 2024-04-22T02:33:47.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2056 16 2502 2486 0.00944539 0 - 0.194117 2024-04-22T02:33:48.456 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2057 16 2502 2486 0.0094408 0 - 0.194117 2024-04-22T02:33:49.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2058 16 2502 2486 0.00943621 0 - 0.194117 2024-04-22T02:33:50.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2059 16 2502 2486 0.00943163 0 - 0.194117 2024-04-22T02:33:51.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:33:51.459381+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:33:51.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:33:51.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2060 16 2502 2486 0.00942705 0 - 0.194117 2024-04-22T02:33:52.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2061 16 2502 2486 0.00942248 0 - 0.194117 2024-04-22T02:33:53.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2062 16 2502 2486 0.00941791 0 - 0.194117 2024-04-22T02:33:54.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2063 16 2502 2486 0.00941334 0 - 0.194117 2024-04-22T02:33:55.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2064 16 2502 2486 0.00940878 0 - 0.194117 2024-04-22T02:33:56.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2065 16 2502 2486 0.00940423 0 - 0.194117 2024-04-22T02:33:57.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2066 16 2502 2486 0.00939968 0 - 0.194117 2024-04-22T02:33:58.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2067 16 2502 2486 0.00939513 0 - 0.194117 2024-04-22T02:33:59.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2068 16 2502 2486 0.00939058 0 - 0.194117 2024-04-22T02:34:00.457 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2069 16 2502 2486 0.00938605 0 - 0.194117 2024-04-22T02:34:01.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2070 16 2502 2486 0.00938151 0 - 0.194117 2024-04-22T02:34:02.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2071 16 2502 2486 0.00937698 0 - 0.194117 2024-04-22T02:34:03.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2072 16 2502 2486 0.00937246 0 - 0.194117 2024-04-22T02:34:04.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2073 16 2502 2486 0.00936794 0 - 0.194117 2024-04-22T02:34:05.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2074 16 2502 2486 0.00936342 0 - 0.194117 2024-04-22T02:34:06.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2075 16 2502 2486 0.00935891 0 - 0.194117 2024-04-22T02:34:07.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2076 16 2502 2486 0.0093544 0 - 0.194117 2024-04-22T02:34:08.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2077 16 2502 2486 0.00934989 0 - 0.194117 2024-04-22T02:34:09.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2078 16 2502 2486 0.0093454 0 - 0.194117 2024-04-22T02:34:10.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2079 16 2502 2486 0.0093409 0 - 0.194117 2024-04-22T02:34:11.458 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:34:11.461123+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:34:11.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:34:11.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2080 16 2502 2486 0.00933641 0 - 0.194117 2024-04-22T02:34:12.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2081 16 2502 2486 0.00933192 0 - 0.194117 2024-04-22T02:34:13.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2082 16 2502 2486 0.00932744 0 - 0.194117 2024-04-22T02:34:14.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2083 16 2502 2486 0.00932296 0 - 0.194117 2024-04-22T02:34:15.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2084 16 2502 2486 0.00931849 0 - 0.194117 2024-04-22T02:34:16.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2085 16 2502 2486 0.00931402 0 - 0.194117 2024-04-22T02:34:17.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2086 16 2502 2486 0.00930956 0 - 0.194117 2024-04-22T02:34:18.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2087 16 2502 2486 0.00930509 0 - 0.194117 2024-04-22T02:34:19.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2088 16 2502 2486 0.00930064 0 - 0.194117 2024-04-22T02:34:20.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2089 16 2502 2486 0.00929619 0 - 0.194117 2024-04-22T02:34:21.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2090 16 2502 2486 0.00929174 0 - 0.194117 2024-04-22T02:34:22.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2091 16 2502 2486 0.0092873 0 - 0.194117 2024-04-22T02:34:23.459 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2092 16 2502 2486 0.00928286 0 - 0.194117 2024-04-22T02:34:24.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2093 16 2502 2486 0.00927842 0 - 0.194117 2024-04-22T02:34:25.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2094 16 2502 2486 0.00927399 0 - 0.194117 2024-04-22T02:34:26.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2095 16 2502 2486 0.00926956 0 - 0.194117 2024-04-22T02:34:27.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2096 16 2502 2486 0.00926514 0 - 0.194117 2024-04-22T02:34:28.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2097 16 2502 2486 0.00926072 0 - 0.194117 2024-04-22T02:34:29.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2098 16 2502 2486 0.00925631 0 - 0.194117 2024-04-22T02:34:30.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2099 16 2502 2486 0.0092519 0 - 0.194117 2024-04-22T02:34:31.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:34:31.463095+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:34:31.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:34:31.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2100 16 2502 2486 0.00924749 0 - 0.194117 2024-04-22T02:34:32.460 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2101 16 2502 2486 0.00924309 0 - 0.194117 2024-04-22T02:34:33.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2102 16 2502 2486 0.00923869 0 - 0.194117 2024-04-22T02:34:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:34:33.933 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:34:34.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2103 16 2502 2486 0.0092343 0 - 0.194117 2024-04-22T02:34:35.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2104 16 2502 2486 0.00922991 0 - 0.194117 2024-04-22T02:34:35.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:34:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:34:36.050+0000 7f28a82ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:34:36.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2105 16 2502 2486 0.00922553 0 - 0.194117 2024-04-22T02:34:37.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2106 16 2502 2486 0.00922115 0 - 0.194117 2024-04-22T02:34:38.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2107 16 2502 2486 0.00921677 0 - 0.194117 2024-04-22T02:34:39.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2108 16 2502 2486 0.0092124 0 - 0.194117 2024-04-22T02:34:40.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2109 16 2502 2486 0.00920803 0 - 0.194117 2024-04-22T02:34:41.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2110 16 2502 2486 0.00920367 0 - 0.194117 2024-04-22T02:34:42.461 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2111 16 2502 2486 0.00919931 0 - 0.194117 2024-04-22T02:34:43.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2112 16 2502 2486 0.00919495 0 - 0.194117 2024-04-22T02:34:44.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2113 16 2502 2486 0.0091906 0 - 0.194117 2024-04-22T02:34:45.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2114 16 2502 2486 0.00918625 0 - 0.194117 2024-04-22T02:34:46.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2115 16 2502 2486 0.00918191 0 - 0.194117 2024-04-22T02:34:47.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2116 16 2502 2486 0.00917757 0 - 0.194117 2024-04-22T02:34:48.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2117 16 2502 2486 0.00917323 0 - 0.194117 2024-04-22T02:34:49.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2118 16 2502 2486 0.0091689 0 - 0.194117 2024-04-22T02:34:50.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2119 16 2502 2486 0.00916458 0 - 0.194117 2024-04-22T02:34:51.462 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:34:51.465187+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:34:51.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:34:51.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2120 16 2502 2486 0.00916025 0 - 0.194117 2024-04-22T02:34:52.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2121 16 2502 2486 0.00915593 0 - 0.194117 2024-04-22T02:34:53.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2122 16 2502 2486 0.00915162 0 - 0.194117 2024-04-22T02:34:54.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2123 16 2502 2486 0.00914731 0 - 0.194117 2024-04-22T02:34:55.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2124 16 2502 2486 0.009143 0 - 0.194117 2024-04-22T02:34:56.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2125 16 2502 2486 0.0091387 0 - 0.194117 2024-04-22T02:34:57.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2126 16 2502 2486 0.0091344 0 - 0.194117 2024-04-22T02:34:58.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2127 16 2502 2486 0.00913011 0 - 0.194117 2024-04-22T02:34:59.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2128 16 2502 2486 0.00912582 0 - 0.194117 2024-04-22T02:35:00.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2129 16 2502 2486 0.00912153 0 - 0.194117 2024-04-22T02:35:01.463 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2130 16 2502 2486 0.00911725 0 - 0.194117 2024-04-22T02:35:02.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2131 16 2502 2486 0.00911297 0 - 0.194117 2024-04-22T02:35:03.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2132 16 2502 2486 0.00910869 0 - 0.194117 2024-04-22T02:35:04.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2133 16 2502 2486 0.00910442 0 - 0.194117 2024-04-22T02:35:05.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2134 16 2502 2486 0.00910016 0 - 0.194117 2024-04-22T02:35:06.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2135 16 2502 2486 0.00909589 0 - 0.194117 2024-04-22T02:35:07.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2136 16 2502 2486 0.00909164 0 - 0.194117 2024-04-22T02:35:08.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2137 16 2502 2486 0.00908738 0 - 0.194117 2024-04-22T02:35:09.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2138 16 2502 2486 0.00908313 0 - 0.194117 2024-04-22T02:35:10.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2139 16 2502 2486 0.00907889 0 - 0.194117 2024-04-22T02:35:11.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:35:11.467203+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:35:11.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:35:11.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2140 16 2502 2486 0.00907464 0 - 0.194117 2024-04-22T02:35:12.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2141 16 2502 2486 0.0090704 0 - 0.194117 2024-04-22T02:35:13.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2142 16 2502 2486 0.00906617 0 - 0.194117 2024-04-22T02:35:14.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2143 16 2502 2486 0.00906194 0 - 0.194117 2024-04-22T02:35:15.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2144 16 2502 2486 0.00905771 0 - 0.194117 2024-04-22T02:35:16.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2145 16 2502 2486 0.00905349 0 - 0.194117 2024-04-22T02:35:17.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2146 16 2502 2486 0.00904927 0 - 0.194117 2024-04-22T02:35:18.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2147 16 2502 2486 0.00904506 0 - 0.194117 2024-04-22T02:35:19.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2148 16 2502 2486 0.00904085 0 - 0.194117 2024-04-22T02:35:20.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2149 16 2502 2486 0.00903664 0 - 0.194117 2024-04-22T02:35:21.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2150 16 2502 2486 0.00903244 0 - 0.194117 2024-04-22T02:35:22.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2151 16 2502 2486 0.00902824 0 - 0.194117 2024-04-22T02:35:23.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2152 16 2502 2486 0.00902404 0 - 0.194117 2024-04-22T02:35:24.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2153 16 2502 2486 0.00901985 0 - 0.194117 2024-04-22T02:35:25.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2154 16 2502 2486 0.00901566 0 - 0.194117 2024-04-22T02:35:26.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2155 16 2502 2486 0.00901148 0 - 0.194117 2024-04-22T02:35:27.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2156 16 2502 2486 0.0090073 0 - 0.194117 2024-04-22T02:35:28.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2157 16 2502 2486 0.00900312 0 - 0.194117 2024-04-22T02:35:29.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2158 16 2502 2486 0.00899895 0 - 0.194117 2024-04-22T02:35:30.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2159 16 2502 2486 0.00899478 0 - 0.194117 2024-04-22T02:35:31.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:35:31.469516+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:35:31.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:35:31.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2160 16 2502 2486 0.00899062 0 - 0.194117 2024-04-22T02:35:32.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2161 16 2502 2486 0.00898646 0 - 0.194117 2024-04-22T02:35:33.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2162 16 2502 2486 0.0089823 0 - 0.194117 2024-04-22T02:35:34.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2163 16 2502 2486 0.00897815 0 - 0.194117 2024-04-22T02:35:35.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2164 16 2502 2486 0.008974 0 - 0.194117 2024-04-22T02:35:36.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2165 16 2502 2486 0.00896986 0 - 0.194117 2024-04-22T02:35:37.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2166 16 2502 2486 0.00896571 0 - 0.194117 2024-04-22T02:35:38.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2167 16 2502 2486 0.00896158 0 - 0.194117 2024-04-22T02:35:39.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2168 16 2502 2486 0.00895744 0 - 0.194117 2024-04-22T02:35:40.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2169 16 2502 2486 0.00895331 0 - 0.194117 2024-04-22T02:35:41.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2170 16 2502 2486 0.00894919 0 - 0.194117 2024-04-22T02:35:42.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2171 16 2502 2486 0.00894506 0 - 0.194117 2024-04-22T02:35:43.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2172 16 2502 2486 0.00894095 0 - 0.194117 2024-04-22T02:35:44.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2173 16 2502 2486 0.00893683 0 - 0.194117 2024-04-22T02:35:45.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2174 16 2502 2486 0.00893272 0 - 0.194117 2024-04-22T02:35:46.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2175 16 2502 2486 0.00892861 0 - 0.194117 2024-04-22T02:35:47.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2176 16 2502 2486 0.00892451 0 - 0.194117 2024-04-22T02:35:48.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2177 16 2502 2486 0.00892041 0 - 0.194117 2024-04-22T02:35:49.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2178 16 2502 2486 0.00891632 0 - 0.194117 2024-04-22T02:35:50.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2179 16 2502 2486 0.00891222 0 - 0.194117 2024-04-22T02:35:51.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:35:51.471593+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:35:51.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:35:51.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2180 16 2502 2486 0.00890814 0 - 0.194117 2024-04-22T02:35:52.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2181 16 2502 2486 0.00890405 0 - 0.194117 2024-04-22T02:35:53.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2182 16 2502 2486 0.00889997 0 - 0.194117 2024-04-22T02:35:54.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2183 16 2502 2486 0.00889589 0 - 0.194117 2024-04-22T02:35:55.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2184 16 2502 2486 0.00889182 0 - 0.194117 2024-04-22T02:35:56.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2185 16 2502 2486 0.00888775 0 - 0.194117 2024-04-22T02:35:57.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2186 16 2502 2486 0.00888369 0 - 0.194117 2024-04-22T02:35:58.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2187 16 2502 2486 0.00887962 0 - 0.194117 2024-04-22T02:35:59.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2188 16 2502 2486 0.00887557 0 - 0.194117 2024-04-22T02:36:00.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2189 16 2502 2486 0.00887151 0 - 0.194117 2024-04-22T02:36:01.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2190 16 2502 2486 0.00886746 0 - 0.194117 2024-04-22T02:36:02.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2191 16 2502 2486 0.00886341 0 - 0.194117 2024-04-22T02:36:03.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2192 16 2502 2486 0.00885937 0 - 0.194117 2024-04-22T02:36:04.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2193 16 2502 2486 0.00885533 0 - 0.194117 2024-04-22T02:36:05.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2194 16 2502 2486 0.00885129 0 - 0.194117 2024-04-22T02:36:06.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2195 16 2502 2486 0.00884726 0 - 0.194117 2024-04-22T02:36:07.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2196 16 2502 2486 0.00884323 0 - 0.194117 2024-04-22T02:36:08.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2197 16 2502 2486 0.00883921 0 - 0.194117 2024-04-22T02:36:09.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2198 16 2502 2486 0.00883519 0 - 0.194117 2024-04-22T02:36:10.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2199 16 2502 2486 0.00883117 0 - 0.194117 2024-04-22T02:36:11.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:36:11.473572+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:36:11.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:36:11.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2200 16 2502 2486 0.00882715 0 - 0.194117 2024-04-22T02:36:12.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2201 16 2502 2486 0.00882314 0 - 0.194117 2024-04-22T02:36:13.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2202 16 2502 2486 0.00881914 0 - 0.194117 2024-04-22T02:36:14.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2203 16 2502 2486 0.00881513 0 - 0.194117 2024-04-22T02:36:15.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2204 16 2502 2486 0.00881113 0 - 0.194117 2024-04-22T02:36:16.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2205 16 2502 2486 0.00880714 0 - 0.194117 2024-04-22T02:36:17.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2206 16 2502 2486 0.00880315 0 - 0.194117 2024-04-22T02:36:18.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2207 16 2502 2486 0.00879916 0 - 0.194117 2024-04-22T02:36:19.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2208 16 2502 2486 0.00879517 0 - 0.194117 2024-04-22T02:36:20.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2209 16 2502 2486 0.00879119 0 - 0.194117 2024-04-22T02:36:21.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2210 16 2502 2486 0.00878721 0 - 0.194117 2024-04-22T02:36:22.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2211 16 2502 2486 0.00878324 0 - 0.194117 2024-04-22T02:36:23.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2212 16 2502 2486 0.00877927 0 - 0.194117 2024-04-22T02:36:24.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2213 16 2502 2486 0.0087753 0 - 0.194117 2024-04-22T02:36:25.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2214 16 2502 2486 0.00877134 0 - 0.194117 2024-04-22T02:36:26.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2215 16 2502 2486 0.00876738 0 - 0.194117 2024-04-22T02:36:27.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2216 16 2502 2486 0.00876342 0 - 0.194117 2024-04-22T02:36:28.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2217 16 2502 2486 0.00875947 0 - 0.194117 2024-04-22T02:36:29.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2218 16 2502 2486 0.00875552 0 - 0.194117 2024-04-22T02:36:30.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2219 16 2502 2486 0.00875157 0 - 0.194117 2024-04-22T02:36:31.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:36:31.475408+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:36:31.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:36:31.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2220 16 2502 2486 0.00874763 0 - 0.194117 2024-04-22T02:36:32.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2221 16 2502 2486 0.00874369 0 - 0.194117 2024-04-22T02:36:33.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2222 16 2502 2486 0.00873976 0 - 0.194117 2024-04-22T02:36:34.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2223 16 2502 2486 0.00873583 0 - 0.194117 2024-04-22T02:36:35.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2224 16 2502 2486 0.0087319 0 - 0.194117 2024-04-22T02:36:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:36:35.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:36:36.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2225 16 2502 2486 0.00872797 0 - 0.194117 2024-04-22T02:36:37.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2226 16 2502 2486 0.00872405 0 - 0.194117 2024-04-22T02:36:37.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:36:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:36:38.085+0000 7f9b8eaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:36:38.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2227 16 2502 2486 0.00872014 0 - 0.194117 2024-04-22T02:36:39.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2228 16 2502 2486 0.00871622 0 - 0.194117 2024-04-22T02:36:40.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2229 16 2502 2486 0.00871231 0 - 0.194117 2024-04-22T02:36:41.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2230 16 2502 2486 0.0087084 0 - 0.194117 2024-04-22T02:36:42.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2231 16 2502 2486 0.0087045 0 - 0.194117 2024-04-22T02:36:43.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2232 16 2502 2486 0.0087006 0 - 0.194117 2024-04-22T02:36:44.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2233 16 2502 2486 0.00869671 0 - 0.194117 2024-04-22T02:36:45.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2234 16 2502 2486 0.00869281 0 - 0.194117 2024-04-22T02:36:46.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2235 16 2502 2486 0.00868892 0 - 0.194117 2024-04-22T02:36:47.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2236 16 2502 2486 0.00868504 0 - 0.194117 2024-04-22T02:36:48.474 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2237 16 2502 2486 0.00868115 0 - 0.194117 2024-04-22T02:36:49.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2238 16 2502 2486 0.00867728 0 - 0.194117 2024-04-22T02:36:50.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2239 16 2502 2486 0.0086734 0 - 0.194117 2024-04-22T02:36:51.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:36:51.477463+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:36:51.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:36:51.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2240 16 2502 2486 0.00866953 0 - 0.194117 2024-04-22T02:36:52.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2241 16 2502 2486 0.00866566 0 - 0.194117 2024-04-22T02:36:53.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2242 16 2502 2486 0.00866179 0 - 0.194117 2024-04-22T02:36:54.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2243 16 2502 2486 0.00865793 0 - 0.194117 2024-04-22T02:36:55.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2244 16 2502 2486 0.00865407 0 - 0.194117 2024-04-22T02:36:56.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2245 16 2502 2486 0.00865022 0 - 0.194117 2024-04-22T02:36:57.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2246 16 2502 2486 0.00864637 0 - 0.194117 2024-04-22T02:36:58.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2247 16 2502 2486 0.00864252 0 - 0.194117 2024-04-22T02:36:59.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2248 16 2502 2486 0.00863868 0 - 0.194117 2024-04-22T02:37:00.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2249 16 2502 2486 0.00863483 0 - 0.194117 2024-04-22T02:37:01.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2250 16 2502 2486 0.008631 0 - 0.194117 2024-04-22T02:37:02.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2251 16 2502 2486 0.00862716 0 - 0.194117 2024-04-22T02:37:03.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2252 16 2502 2486 0.00862333 0 - 0.194117 2024-04-22T02:37:04.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2253 16 2502 2486 0.0086195 0 - 0.194117 2024-04-22T02:37:05.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2254 16 2502 2486 0.00861568 0 - 0.194117 2024-04-22T02:37:06.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2255 16 2502 2486 0.00861186 0 - 0.194117 2024-04-22T02:37:07.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2256 16 2502 2486 0.00860804 0 - 0.194117 2024-04-22T02:37:08.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2257 16 2502 2486 0.00860423 0 - 0.194117 2024-04-22T02:37:09.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2258 16 2502 2486 0.00860042 0 - 0.194117 2024-04-22T02:37:10.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2259 16 2502 2486 0.00859661 0 - 0.194117 2024-04-22T02:37:11.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:37:11.479730+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:37:11.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:37:11.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2260 16 2502 2486 0.00859281 0 - 0.194117 2024-04-22T02:37:12.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2261 16 2502 2486 0.00858901 0 - 0.194117 2024-04-22T02:37:13.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2262 16 2502 2486 0.00858521 0 - 0.194117 2024-04-22T02:37:14.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2263 16 2502 2486 0.00858142 0 - 0.194117 2024-04-22T02:37:15.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2264 16 2502 2486 0.00857763 0 - 0.194117 2024-04-22T02:37:16.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2265 16 2502 2486 0.00857384 0 - 0.194117 2024-04-22T02:37:17.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2266 16 2502 2486 0.00857005 0 - 0.194117 2024-04-22T02:37:18.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2267 16 2502 2486 0.00856627 0 - 0.194117 2024-04-22T02:37:19.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2268 16 2502 2486 0.0085625 0 - 0.194117 2024-04-22T02:37:20.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2269 16 2502 2486 0.00855872 0 - 0.194117 2024-04-22T02:37:21.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2270 16 2502 2486 0.00855495 0 - 0.194117 2024-04-22T02:37:22.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2271 16 2502 2486 0.00855119 0 - 0.194117 2024-04-22T02:37:23.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2272 16 2502 2486 0.00854742 0 - 0.194117 2024-04-22T02:37:24.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2273 16 2502 2486 0.00854366 0 - 0.194117 2024-04-22T02:37:25.478 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2274 16 2502 2486 0.00853991 0 - 0.194117 2024-04-22T02:37:26.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2275 16 2502 2486 0.00853615 0 - 0.194117 2024-04-22T02:37:27.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2276 16 2502 2486 0.0085324 0 - 0.194117 2024-04-22T02:37:28.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2277 16 2502 2486 0.00852865 0 - 0.194117 2024-04-22T02:37:29.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2278 16 2502 2486 0.00852491 0 - 0.194117 2024-04-22T02:37:30.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2279 16 2502 2486 0.00852117 0 - 0.194117 2024-04-22T02:37:31.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:37:31.481696+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:37:31.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:37:31.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2280 16 2502 2486 0.00851743 0 - 0.194117 2024-04-22T02:37:32.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2281 16 2502 2486 0.0085137 0 - 0.194117 2024-04-22T02:37:33.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2282 16 2502 2486 0.00850997 0 - 0.194117 2024-04-22T02:37:34.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2283 16 2502 2486 0.00850624 0 - 0.194117 2024-04-22T02:37:35.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2284 16 2502 2486 0.00850252 0 - 0.194117 2024-04-22T02:37:36.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2285 16 2502 2486 0.00849879 0 - 0.194117 2024-04-22T02:37:37.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2286 16 2502 2486 0.00849508 0 - 0.194117 2024-04-22T02:37:38.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2287 16 2502 2486 0.00849136 0 - 0.194117 2024-04-22T02:37:39.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2288 16 2502 2486 0.00848765 0 - 0.194117 2024-04-22T02:37:40.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2289 16 2502 2486 0.00848394 0 - 0.194117 2024-04-22T02:37:41.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2290 16 2502 2486 0.00848024 0 - 0.194117 2024-04-22T02:37:42.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2291 16 2502 2486 0.00847654 0 - 0.194117 2024-04-22T02:37:43.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2292 16 2502 2486 0.00847284 0 - 0.194117 2024-04-22T02:37:44.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2293 16 2502 2486 0.00846914 0 - 0.194117 2024-04-22T02:37:45.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2294 16 2502 2486 0.00846545 0 - 0.194117 2024-04-22T02:37:46.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2295 16 2502 2486 0.00846176 0 - 0.194117 2024-04-22T02:37:47.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2296 16 2502 2486 0.00845808 0 - 0.194117 2024-04-22T02:37:48.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2297 16 2502 2486 0.0084544 0 - 0.194117 2024-04-22T02:37:49.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2298 16 2502 2486 0.00845072 0 - 0.194117 2024-04-22T02:37:50.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2299 16 2502 2486 0.00844704 0 - 0.194117 2024-04-22T02:37:51.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:37:51.483712+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:37:51.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:37:51.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2300 16 2502 2486 0.00844337 0 - 0.194117 2024-04-22T02:37:52.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2301 16 2502 2486 0.0084397 0 - 0.194117 2024-04-22T02:37:53.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2302 16 2502 2486 0.00843603 0 - 0.194117 2024-04-22T02:37:54.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2303 16 2502 2486 0.00843237 0 - 0.194117 2024-04-22T02:37:55.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2304 16 2502 2486 0.00842871 0 - 0.194117 2024-04-22T02:37:56.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2305 16 2502 2486 0.00842505 0 - 0.194117 2024-04-22T02:37:57.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2306 16 2502 2486 0.0084214 0 - 0.194117 2024-04-22T02:37:58.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2307 16 2502 2486 0.00841775 0 - 0.194117 2024-04-22T02:37:59.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2308 16 2502 2486 0.0084141 0 - 0.194117 2024-04-22T02:38:00.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2309 16 2502 2486 0.00841046 0 - 0.194117 2024-04-22T02:38:01.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2310 16 2502 2486 0.00840682 0 - 0.194117 2024-04-22T02:38:02.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2311 16 2502 2486 0.00840318 0 - 0.194117 2024-04-22T02:38:03.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2312 16 2502 2486 0.00839955 0 - 0.194117 2024-04-22T02:38:04.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2313 16 2502 2486 0.00839591 0 - 0.194117 2024-04-22T02:38:05.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2314 16 2502 2486 0.00839229 0 - 0.194117 2024-04-22T02:38:06.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2315 16 2502 2486 0.00838866 0 - 0.194117 2024-04-22T02:38:07.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2316 16 2502 2486 0.00838504 0 - 0.194117 2024-04-22T02:38:08.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2317 16 2502 2486 0.00838142 0 - 0.194117 2024-04-22T02:38:09.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2318 16 2502 2486 0.0083778 0 - 0.194117 2024-04-22T02:38:10.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2319 16 2502 2486 0.00837419 0 - 0.194117 2024-04-22T02:38:11.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:38:11.485736+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:38:11.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:38:11.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2320 16 2502 2486 0.00837058 0 - 0.194117 2024-04-22T02:38:12.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2321 16 2502 2486 0.00836697 0 - 0.194117 2024-04-22T02:38:13.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2322 16 2502 2486 0.00836337 0 - 0.194117 2024-04-22T02:38:14.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2323 16 2502 2486 0.00835977 0 - 0.194117 2024-04-22T02:38:15.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2324 16 2502 2486 0.00835617 0 - 0.194117 2024-04-22T02:38:16.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2325 16 2502 2486 0.00835258 0 - 0.194117 2024-04-22T02:38:17.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2326 16 2502 2486 0.00834899 0 - 0.194117 2024-04-22T02:38:18.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2327 16 2502 2486 0.0083454 0 - 0.194117 2024-04-22T02:38:19.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2328 16 2502 2486 0.00834182 0 - 0.194117 2024-04-22T02:38:20.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2329 16 2502 2486 0.00833823 0 - 0.194117 2024-04-22T02:38:21.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2330 16 2502 2486 0.00833466 0 - 0.194117 2024-04-22T02:38:22.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2331 16 2502 2486 0.00833108 0 - 0.194117 2024-04-22T02:38:23.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2332 16 2502 2486 0.00832751 0 - 0.194117 2024-04-22T02:38:24.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2333 16 2502 2486 0.00832394 0 - 0.194117 2024-04-22T02:38:25.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2334 16 2502 2486 0.00832037 0 - 0.194117 2024-04-22T02:38:26.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2335 16 2502 2486 0.00831681 0 - 0.194117 2024-04-22T02:38:27.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2336 16 2502 2486 0.00831325 0 - 0.194117 2024-04-22T02:38:28.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2337 16 2502 2486 0.00830969 0 - 0.194117 2024-04-22T02:38:29.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2338 16 2502 2486 0.00830614 0 - 0.194117 2024-04-22T02:38:30.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2339 16 2502 2486 0.00830259 0 - 0.194117 2024-04-22T02:38:31.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:38:31.487637+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:38:31.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:38:31.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2340 16 2502 2486 0.00829904 0 - 0.194117 2024-04-22T02:38:32.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2341 16 2502 2486 0.00829549 0 - 0.194117 2024-04-22T02:38:33.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2342 16 2502 2486 0.00829195 0 - 0.194117 2024-04-22T02:38:34.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2343 16 2502 2486 0.00828841 0 - 0.194117 2024-04-22T02:38:35.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2344 16 2502 2486 0.00828488 0 - 0.194117 2024-04-22T02:38:36.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2345 16 2502 2486 0.00828134 0 - 0.194117 2024-04-22T02:38:37.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2346 16 2502 2486 0.00827781 0 - 0.194117 2024-04-22T02:38:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:38:38.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:38:38.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2347 16 2502 2486 0.00827429 0 - 0.194117 2024-04-22T02:38:39.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2348 16 2502 2486 0.00827076 0 - 0.194117 2024-04-22T02:38:40.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:38:40.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:38:40.123+0000 7f2044b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:38:40.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2349 16 2502 2486 0.00826724 0 - 0.194117 2024-04-22T02:38:41.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2350 16 2502 2486 0.00826372 0 - 0.194117 2024-04-22T02:38:42.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2351 16 2502 2486 0.00826021 0 - 0.194117 2024-04-22T02:38:43.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2352 16 2502 2486 0.0082567 0 - 0.194117 2024-04-22T02:38:44.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2353 16 2502 2486 0.00825319 0 - 0.194117 2024-04-22T02:38:45.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2354 16 2502 2486 0.00824968 0 - 0.194117 2024-04-22T02:38:46.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2355 16 2502 2486 0.00824618 0 - 0.194117 2024-04-22T02:38:47.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2356 16 2502 2486 0.00824268 0 - 0.194117 2024-04-22T02:38:48.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2357 16 2502 2486 0.00823918 0 - 0.194117 2024-04-22T02:38:49.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2358 16 2502 2486 0.00823569 0 - 0.194117 2024-04-22T02:38:50.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2359 16 2502 2486 0.0082322 0 - 0.194117 2024-04-22T02:38:51.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:38:51.489959+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:38:51.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:38:51.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2360 16 2502 2486 0.00822871 0 - 0.194117 2024-04-22T02:38:52.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2361 16 2502 2486 0.00822522 0 - 0.194117 2024-04-22T02:38:53.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2362 16 2502 2486 0.00822174 0 - 0.194117 2024-04-22T02:38:54.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2363 16 2502 2486 0.00821826 0 - 0.194117 2024-04-22T02:38:55.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2364 16 2502 2486 0.00821478 0 - 0.194117 2024-04-22T02:38:56.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2365 16 2502 2486 0.00821131 0 - 0.194117 2024-04-22T02:38:57.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2366 16 2502 2486 0.00820784 0 - 0.194117 2024-04-22T02:38:58.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2367 16 2502 2486 0.00820437 0 - 0.194117 2024-04-22T02:38:59.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2368 16 2502 2486 0.00820091 0 - 0.194117 2024-04-22T02:39:00.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2369 16 2502 2486 0.00819745 0 - 0.194117 2024-04-22T02:39:01.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2370 16 2502 2486 0.00819399 0 - 0.194117 2024-04-22T02:39:02.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2371 16 2502 2486 0.00819053 0 - 0.194117 2024-04-22T02:39:03.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2372 16 2502 2486 0.00818708 0 - 0.194117 2024-04-22T02:39:04.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2373 16 2502 2486 0.00818363 0 - 0.194117 2024-04-22T02:39:05.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2374 16 2502 2486 0.00818018 0 - 0.194117 2024-04-22T02:39:06.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2375 16 2502 2486 0.00817674 0 - 0.194117 2024-04-22T02:39:07.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2376 16 2502 2486 0.00817329 0 - 0.194117 2024-04-22T02:39:08.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2377 16 2502 2486 0.00816986 0 - 0.194117 2024-04-22T02:39:09.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2378 16 2502 2486 0.00816642 0 - 0.194117 2024-04-22T02:39:10.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2379 16 2502 2486 0.00816299 0 - 0.194117 2024-04-22T02:39:11.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:39:11.492259+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:39:11.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:39:11.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2380 16 2502 2486 0.00815956 0 - 0.194117 2024-04-22T02:39:12.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2381 16 2502 2486 0.00815613 0 - 0.194117 2024-04-22T02:39:13.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2382 16 2502 2486 0.00815271 0 - 0.194117 2024-04-22T02:39:14.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2383 16 2502 2486 0.00814929 0 - 0.194117 2024-04-22T02:39:15.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2384 16 2502 2486 0.00814587 0 - 0.194117 2024-04-22T02:39:16.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2385 16 2502 2486 0.00814245 0 - 0.194117 2024-04-22T02:39:17.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2386 16 2502 2486 0.00813904 0 - 0.194117 2024-04-22T02:39:18.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2387 16 2502 2486 0.00813563 0 - 0.194117 2024-04-22T02:39:19.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2388 16 2502 2486 0.00813222 0 - 0.194117 2024-04-22T02:39:20.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2389 16 2502 2486 0.00812882 0 - 0.194117 2024-04-22T02:39:21.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2390 16 2502 2486 0.00812542 0 - 0.194117 2024-04-22T02:39:22.490 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2391 16 2502 2486 0.00812202 0 - 0.194117 2024-04-22T02:39:23.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2392 16 2502 2486 0.00811862 0 - 0.194117 2024-04-22T02:39:24.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2393 16 2502 2486 0.00811523 0 - 0.194117 2024-04-22T02:39:25.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2394 16 2502 2486 0.00811184 0 - 0.194117 2024-04-22T02:39:26.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2395 16 2502 2486 0.00810845 0 - 0.194117 2024-04-22T02:39:27.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2396 16 2502 2486 0.00810507 0 - 0.194117 2024-04-22T02:39:28.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2397 16 2502 2486 0.00810169 0 - 0.194117 2024-04-22T02:39:29.491 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2398 16 2502 2486 0.00809831 0 - 0.194117 2024-04-22T02:39:30.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2399 16 2502 2486 0.00809493 0 - 0.194117 2024-04-22T02:39:31.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:39:31.494524+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:39:31.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:39:31.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2400 16 2502 2486 0.00809156 0 - 0.194117 2024-04-22T02:39:32.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2401 16 2502 2486 0.00808819 0 - 0.194117 2024-04-22T02:39:33.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2402 16 2502 2486 0.00808482 0 - 0.194117 2024-04-22T02:39:34.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2403 16 2502 2486 0.00808146 0 - 0.194117 2024-04-22T02:39:35.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2404 16 2502 2486 0.0080781 0 - 0.194117 2024-04-22T02:39:36.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2405 16 2502 2486 0.00807474 0 - 0.194117 2024-04-22T02:39:37.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2406 16 2502 2486 0.00807138 0 - 0.194117 2024-04-22T02:39:38.492 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2407 16 2502 2486 0.00806803 0 - 0.194117 2024-04-22T02:39:39.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2408 16 2502 2486 0.00806468 0 - 0.194117 2024-04-22T02:39:40.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2409 16 2502 2486 0.00806133 0 - 0.194117 2024-04-22T02:39:41.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2410 16 2502 2486 0.00805799 0 - 0.194117 2024-04-22T02:39:42.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2411 16 2502 2486 0.00805464 0 - 0.194117 2024-04-22T02:39:43.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2412 16 2502 2486 0.00805131 0 - 0.194117 2024-04-22T02:39:44.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2413 16 2502 2486 0.00804797 0 - 0.194117 2024-04-22T02:39:45.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2414 16 2502 2486 0.00804463 0 - 0.194117 2024-04-22T02:39:46.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2415 16 2502 2486 0.0080413 0 - 0.194117 2024-04-22T02:39:47.493 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2416 16 2502 2486 0.00803798 0 - 0.194117 2024-04-22T02:39:48.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2417 16 2502 2486 0.00803465 0 - 0.194117 2024-04-22T02:39:49.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2418 16 2502 2486 0.00803133 0 - 0.194117 2024-04-22T02:39:50.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2419 16 2502 2486 0.00802801 0 - 0.194117 2024-04-22T02:39:51.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:39:51.496783+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:39:51.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:39:51.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2420 16 2502 2486 0.00802469 0 - 0.194117 2024-04-22T02:39:52.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2421 16 2502 2486 0.00802137 0 - 0.194117 2024-04-22T02:39:53.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2422 16 2502 2486 0.00801806 0 - 0.194117 2024-04-22T02:39:54.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2423 16 2502 2486 0.00801475 0 - 0.194117 2024-04-22T02:39:55.494 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2424 16 2502 2486 0.00801145 0 - 0.194117 2024-04-22T02:39:56.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2425 16 2502 2486 0.00800814 0 - 0.194117 2024-04-22T02:39:57.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2426 16 2502 2486 0.00800484 0 - 0.194117 2024-04-22T02:39:58.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2427 16 2502 2486 0.00800154 0 - 0.194117 2024-04-22T02:39:59.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2428 16 2502 2486 0.00799825 0 - 0.194117 2024-04-22T02:40:00.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2429 16 2502 2486 0.00799496 0 - 0.194117 2024-04-22T02:40:01.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2430 16 2502 2486 0.00799167 0 - 0.194117 2024-04-22T02:40:02.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2431 16 2502 2486 0.00798838 0 - 0.194117 2024-04-22T02:40:03.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2432 16 2502 2486 0.00798509 0 - 0.194117 2024-04-22T02:40:04.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2433 16 2502 2486 0.00798181 0 - 0.194117 2024-04-22T02:40:05.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2434 16 2502 2486 0.00797853 0 - 0.194117 2024-04-22T02:40:06.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2435 16 2502 2486 0.00797526 0 - 0.194117 2024-04-22T02:40:07.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2436 16 2502 2486 0.00797198 0 - 0.194117 2024-04-22T02:40:08.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2437 16 2502 2486 0.00796871 0 - 0.194117 2024-04-22T02:40:09.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2438 16 2502 2486 0.00796544 0 - 0.194117 2024-04-22T02:40:10.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2439 16 2502 2486 0.00796218 0 - 0.194117 2024-04-22T02:40:11.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:40:11.499030+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:40:11.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:40:11.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2440 16 2502 2486 0.00795891 0 - 0.194117 2024-04-22T02:40:12.496 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2441 16 2502 2486 0.00795565 0 - 0.194117 2024-04-22T02:40:13.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2442 16 2502 2486 0.00795239 0 - 0.194117 2024-04-22T02:40:14.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2443 16 2502 2486 0.00794914 0 - 0.194117 2024-04-22T02:40:15.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2444 16 2502 2486 0.00794589 0 - 0.194117 2024-04-22T02:40:16.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2445 16 2502 2486 0.00794264 0 - 0.194117 2024-04-22T02:40:17.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2446 16 2502 2486 0.00793939 0 - 0.194117 2024-04-22T02:40:18.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2447 16 2502 2486 0.00793615 0 - 0.194117 2024-04-22T02:40:19.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2448 16 2502 2486 0.0079329 0 - 0.194117 2024-04-22T02:40:20.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2449 16 2502 2486 0.00792966 0 - 0.194117 2024-04-22T02:40:21.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2450 16 2502 2486 0.00792643 0 - 0.194117 2024-04-22T02:40:22.497 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2451 16 2502 2486 0.00792319 0 - 0.194117 2024-04-22T02:40:23.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2452 16 2502 2486 0.00791996 0 - 0.194117 2024-04-22T02:40:24.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2453 16 2502 2486 0.00791673 0 - 0.194117 2024-04-22T02:40:25.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2454 16 2502 2486 0.00791351 0 - 0.194117 2024-04-22T02:40:26.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2455 16 2502 2486 0.00791028 0 - 0.194117 2024-04-22T02:40:27.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2456 16 2502 2486 0.00790706 0 - 0.194117 2024-04-22T02:40:28.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2457 16 2502 2486 0.00790384 0 - 0.194117 2024-04-22T02:40:29.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2458 16 2502 2486 0.00790063 0 - 0.194117 2024-04-22T02:40:30.498 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2459 16 2502 2486 0.00789742 0 - 0.194117 2024-04-22T02:40:31.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:40:31.501306+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:40:31.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:40:31.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2460 16 2502 2486 0.00789421 0 - 0.194117 2024-04-22T02:40:32.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2461 16 2502 2486 0.007891 0 - 0.194117 2024-04-22T02:40:33.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2462 16 2502 2486 0.00788779 0 - 0.194117 2024-04-22T02:40:34.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2463 16 2502 2486 0.00788459 0 - 0.194117 2024-04-22T02:40:35.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2464 16 2502 2486 0.00788139 0 - 0.194117 2024-04-22T02:40:36.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2465 16 2502 2486 0.00787819 0 - 0.194117 2024-04-22T02:40:37.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2466 16 2502 2486 0.007875 0 - 0.194117 2024-04-22T02:40:38.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2467 16 2502 2486 0.00787181 0 - 0.194117 2024-04-22T02:40:39.499 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2468 16 2502 2486 0.00786862 0 - 0.194117 2024-04-22T02:40:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:40:40.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:40:40.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2469 16 2502 2486 0.00786543 0 - 0.194117 2024-04-22T02:40:41.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2470 16 2502 2486 0.00786225 0 - 0.194117 2024-04-22T02:40:42.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:40:42.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:40:42.161+0000 7f2a1591d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:40:42.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2471 16 2502 2486 0.00785906 0 - 0.194117 2024-04-22T02:40:43.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2472 16 2502 2486 0.00785588 0 - 0.194117 2024-04-22T02:40:44.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2473 16 2502 2486 0.00785271 0 - 0.194117 2024-04-22T02:40:45.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2474 16 2502 2486 0.00784953 0 - 0.194117 2024-04-22T02:40:46.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2475 16 2502 2486 0.00784636 0 - 0.194117 2024-04-22T02:40:47.500 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2476 16 2502 2486 0.00784319 0 - 0.194117 2024-04-22T02:40:48.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2477 16 2502 2486 0.00784003 0 - 0.194117 2024-04-22T02:40:49.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2478 16 2502 2486 0.00783686 0 - 0.194117 2024-04-22T02:40:50.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2479 16 2502 2486 0.0078337 0 - 0.194117 2024-04-22T02:40:51.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:40:51.503704+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:40:51.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:40:51.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2480 16 2502 2486 0.00783054 0 - 0.194117 2024-04-22T02:40:52.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2481 16 2502 2486 0.00782739 0 - 0.194117 2024-04-22T02:40:53.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2482 16 2502 2486 0.00782423 0 - 0.194117 2024-04-22T02:40:54.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2483 16 2502 2486 0.00782108 0 - 0.194117 2024-04-22T02:40:55.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2484 16 2502 2486 0.00781793 0 - 0.194117 2024-04-22T02:40:56.501 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2485 16 2502 2486 0.00781479 0 - 0.194117 2024-04-22T02:40:57.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2486 16 2502 2486 0.00781164 0 - 0.194117 2024-04-22T02:40:58.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2487 16 2502 2486 0.0078085 0 - 0.194117 2024-04-22T02:40:59.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2488 16 2502 2486 0.00780536 0 - 0.194117 2024-04-22T02:41:00.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2489 16 2502 2486 0.00780223 0 - 0.194117 2024-04-22T02:41:01.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2490 16 2502 2486 0.00779909 0 - 0.194117 2024-04-22T02:41:02.502 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2491 16 2502 2486 0.00779596 0 - 0.194117 2024-04-22T02:41:03.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2492 16 2502 2486 0.00779283 0 - 0.194117 2024-04-22T02:41:04.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2493 16 2502 2486 0.00778971 0 - 0.194117 2024-04-22T02:41:05.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2494 16 2502 2486 0.00778658 0 - 0.194117 2024-04-22T02:41:06.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2495 16 2502 2486 0.00778346 0 - 0.194117 2024-04-22T02:41:07.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2496 16 2502 2486 0.00778035 0 - 0.194117 2024-04-22T02:41:08.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2497 16 2502 2486 0.00777723 0 - 0.194117 2024-04-22T02:41:09.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2498 16 2502 2486 0.00777412 0 - 0.194117 2024-04-22T02:41:10.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2499 16 2502 2486 0.00777101 0 - 0.194117 2024-04-22T02:41:11.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:41:11.506139+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:41:11.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:41:11.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2500 16 2502 2486 0.0077679 0 - 0.194117 2024-04-22T02:41:12.503 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2501 16 2502 2486 0.00776479 0 - 0.194117 2024-04-22T02:41:13.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2502 16 2502 2486 0.00776169 0 - 0.194117 2024-04-22T02:41:14.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2503 16 2502 2486 0.00775859 0 - 0.194117 2024-04-22T02:41:15.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2504 16 2502 2486 0.00775549 0 - 0.194117 2024-04-22T02:41:16.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2505 16 2502 2486 0.00775239 0 - 0.194117 2024-04-22T02:41:17.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2506 16 2502 2486 0.0077493 0 - 0.194117 2024-04-22T02:41:18.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2507 16 2502 2486 0.00774621 0 - 0.194117 2024-04-22T02:41:19.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2508 16 2502 2486 0.00774312 0 - 0.194117 2024-04-22T02:41:20.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2509 16 2502 2486 0.00774003 0 - 0.194117 2024-04-22T02:41:21.504 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2510 16 2502 2486 0.00773695 0 - 0.194117 2024-04-22T02:41:22.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2511 16 2502 2486 0.00773387 0 - 0.194117 2024-04-22T02:41:23.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2512 16 2502 2486 0.00773079 0 - 0.194117 2024-04-22T02:41:24.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2513 16 2502 2486 0.00772771 0 - 0.194117 2024-04-22T02:41:25.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2514 16 2502 2486 0.00772464 0 - 0.194117 2024-04-22T02:41:26.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2515 16 2502 2486 0.00772157 0 - 0.194117 2024-04-22T02:41:27.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2516 16 2502 2486 0.0077185 0 - 0.194117 2024-04-22T02:41:28.505 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2517 16 2502 2486 0.00771543 0 - 0.194117 2024-04-22T02:41:29.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2518 16 2502 2486 0.00771237 0 - 0.194117 2024-04-22T02:41:30.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2519 16 2502 2486 0.00770931 0 - 0.194117 2024-04-22T02:41:31.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:41:31.508535+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:41:31.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:41:31.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2520 16 2502 2486 0.00770625 0 - 0.194117 2024-04-22T02:41:32.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2521 16 2502 2486 0.00770319 0 - 0.194117 2024-04-22T02:41:33.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2522 16 2502 2486 0.00770014 0 - 0.194117 2024-04-22T02:41:34.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2523 16 2502 2486 0.00769708 0 - 0.194117 2024-04-22T02:41:35.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2524 16 2502 2486 0.00769403 0 - 0.194117 2024-04-22T02:41:36.506 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2525 16 2502 2486 0.00769099 0 - 0.194117 2024-04-22T02:41:37.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2526 16 2502 2486 0.00768794 0 - 0.194117 2024-04-22T02:41:38.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2527 16 2502 2486 0.0076849 0 - 0.194117 2024-04-22T02:41:39.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2528 16 2502 2486 0.00768186 0 - 0.194117 2024-04-22T02:41:40.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2529 16 2502 2486 0.00767882 0 - 0.194117 2024-04-22T02:41:41.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2530 16 2502 2486 0.00767579 0 - 0.194117 2024-04-22T02:41:42.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2531 16 2502 2486 0.00767275 0 - 0.194117 2024-04-22T02:41:43.507 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2532 16 2502 2486 0.00766972 0 - 0.194117 2024-04-22T02:41:44.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2533 16 2502 2486 0.0076667 0 - 0.194117 2024-04-22T02:41:45.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2534 16 2502 2486 0.00766367 0 - 0.194117 2024-04-22T02:41:46.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2535 16 2502 2486 0.00766065 0 - 0.194117 2024-04-22T02:41:47.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2536 16 2502 2486 0.00765763 0 - 0.194117 2024-04-22T02:41:48.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2537 16 2502 2486 0.00765461 0 - 0.194117 2024-04-22T02:41:49.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2538 16 2502 2486 0.00765159 0 - 0.194117 2024-04-22T02:41:50.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2539 16 2502 2486 0.00764858 0 - 0.194117 2024-04-22T02:41:51.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:41:51.511121+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:41:51.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:41:51.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2540 16 2502 2486 0.00764557 0 - 0.194117 2024-04-22T02:41:52.508 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2541 16 2502 2486 0.00764256 0 - 0.194117 2024-04-22T02:41:53.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2542 16 2502 2486 0.00763955 0 - 0.194117 2024-04-22T02:41:54.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2543 16 2502 2486 0.00763655 0 - 0.194117 2024-04-22T02:41:55.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2544 16 2502 2486 0.00763354 0 - 0.194117 2024-04-22T02:41:56.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2545 16 2502 2486 0.00763055 0 - 0.194117 2024-04-22T02:41:57.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2546 16 2502 2486 0.00762755 0 - 0.194117 2024-04-22T02:41:58.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2547 16 2502 2486 0.00762455 0 - 0.194117 2024-04-22T02:41:59.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2548 16 2502 2486 0.00762156 0 - 0.194117 2024-04-22T02:42:00.509 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2549 16 2502 2486 0.00761857 0 - 0.194117 2024-04-22T02:42:01.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2550 16 2502 2486 0.00761558 0 - 0.194117 2024-04-22T02:42:02.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2551 16 2502 2486 0.0076126 0 - 0.194117 2024-04-22T02:42:03.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2552 16 2502 2486 0.00760961 0 - 0.194117 2024-04-22T02:42:04.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2553 16 2502 2486 0.00760663 0 - 0.194117 2024-04-22T02:42:05.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2554 16 2502 2486 0.00760366 0 - 0.194117 2024-04-22T02:42:06.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2555 16 2502 2486 0.00760068 0 - 0.194117 2024-04-22T02:42:07.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2556 16 2502 2486 0.00759771 0 - 0.194117 2024-04-22T02:42:08.510 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2557 16 2502 2486 0.00759473 0 - 0.194117 2024-04-22T02:42:09.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2558 16 2502 2486 0.00759177 0 - 0.194117 2024-04-22T02:42:10.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2559 16 2502 2486 0.0075888 0 - 0.194117 2024-04-22T02:42:11.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:42:11.513655+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:42:11.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:42:11.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2560 16 2502 2486 0.00758583 0 - 0.194117 2024-04-22T02:42:12.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2561 16 2502 2486 0.00758287 0 - 0.194117 2024-04-22T02:42:13.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2562 16 2502 2486 0.00757991 0 - 0.194117 2024-04-22T02:42:14.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2563 16 2502 2486 0.00757695 0 - 0.194117 2024-04-22T02:42:15.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2564 16 2502 2486 0.007574 0 - 0.194117 2024-04-22T02:42:16.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2565 16 2502 2486 0.00757105 0 - 0.194117 2024-04-22T02:42:17.511 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2566 16 2502 2486 0.0075681 0 - 0.194117 2024-04-22T02:42:18.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2567 16 2502 2486 0.00756515 0 - 0.194117 2024-04-22T02:42:19.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2568 16 2502 2486 0.0075622 0 - 0.194117 2024-04-22T02:42:20.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2569 16 2502 2486 0.00755926 0 - 0.194117 2024-04-22T02:42:21.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2570 16 2502 2486 0.00755632 0 - 0.194117 2024-04-22T02:42:22.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2571 16 2502 2486 0.00755338 0 - 0.194117 2024-04-22T02:42:23.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2572 16 2502 2486 0.00755044 0 - 0.194117 2024-04-22T02:42:24.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2573 16 2502 2486 0.00754751 0 - 0.194117 2024-04-22T02:42:25.512 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2574 16 2502 2486 0.00754457 0 - 0.194117 2024-04-22T02:42:26.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2575 16 2502 2486 0.00754164 0 - 0.194117 2024-04-22T02:42:27.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2576 16 2502 2486 0.00753872 0 - 0.194117 2024-04-22T02:42:28.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2577 16 2502 2486 0.00753579 0 - 0.194117 2024-04-22T02:42:29.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2578 16 2502 2486 0.00753287 0 - 0.194117 2024-04-22T02:42:30.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2579 16 2502 2486 0.00752995 0 - 0.194117 2024-04-22T02:42:31.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:42:31.515875+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:42:31.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:42:31.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2580 16 2502 2486 0.00752703 0 - 0.194117 2024-04-22T02:42:32.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2581 16 2502 2486 0.00752411 0 - 0.194117 2024-04-22T02:42:33.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2582 16 2502 2486 0.0075212 0 - 0.194117 2024-04-22T02:42:34.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2583 16 2502 2486 0.00751829 0 - 0.194117 2024-04-22T02:42:35.513 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2584 16 2502 2486 0.00751538 0 - 0.194117 2024-04-22T02:42:36.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2585 16 2502 2486 0.00751247 0 - 0.194117 2024-04-22T02:42:37.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2586 16 2502 2486 0.00750957 0 - 0.194117 2024-04-22T02:42:38.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2587 16 2502 2486 0.00750666 0 - 0.194117 2024-04-22T02:42:39.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2588 16 2502 2486 0.00750376 0 - 0.194117 2024-04-22T02:42:40.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2589 16 2502 2486 0.00750086 0 - 0.194117 2024-04-22T02:42:41.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2590 16 2502 2486 0.00749797 0 - 0.194117 2024-04-22T02:42:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:42:42.081 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:42:42.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2591 16 2502 2486 0.00749507 0 - 0.194117 2024-04-22T02:42:43.514 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2592 16 2502 2486 0.00749218 0 - 0.194117 2024-04-22T02:42:44.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:42:44.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:42:44.201+0000 7fc849d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:42:44.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2593 16 2502 2486 0.00748929 0 - 0.194117 2024-04-22T02:42:45.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2594 16 2502 2486 0.00748641 0 - 0.194117 2024-04-22T02:42:46.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2595 16 2502 2486 0.00748352 0 - 0.194117 2024-04-22T02:42:47.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2596 16 2502 2486 0.00748064 0 - 0.194117 2024-04-22T02:42:48.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2597 16 2502 2486 0.00747776 0 - 0.194117 2024-04-22T02:42:49.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2598 16 2502 2486 0.00747488 0 - 0.194117 2024-04-22T02:42:50.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2599 16 2502 2486 0.007472 0 - 0.194117 2024-04-22T02:42:51.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:42:51.518185+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:42:51.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:42:51.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2600 16 2502 2486 0.00746913 0 - 0.194117 2024-04-22T02:42:52.515 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2601 16 2502 2486 0.00746626 0 - 0.194117 2024-04-22T02:42:53.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2602 16 2502 2486 0.00746339 0 - 0.194117 2024-04-22T02:42:54.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2603 16 2502 2486 0.00746052 0 - 0.194117 2024-04-22T02:42:55.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2604 16 2502 2486 0.00745766 0 - 0.194117 2024-04-22T02:42:56.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2605 16 2502 2486 0.00745479 0 - 0.194117 2024-04-22T02:42:57.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2606 16 2502 2486 0.00745193 0 - 0.194117 2024-04-22T02:42:58.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2607 16 2502 2486 0.00744907 0 - 0.194117 2024-04-22T02:42:59.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2608 16 2502 2486 0.00744622 0 - 0.194117 2024-04-22T02:43:00.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2609 16 2502 2486 0.00744336 0 - 0.194117 2024-04-22T02:43:01.516 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2610 16 2502 2486 0.00744051 0 - 0.194117 2024-04-22T02:43:02.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2611 16 2502 2486 0.00743766 0 - 0.194117 2024-04-22T02:43:03.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2612 16 2502 2486 0.00743481 0 - 0.194117 2024-04-22T02:43:04.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2613 16 2502 2486 0.00743197 0 - 0.194117 2024-04-22T02:43:05.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2614 16 2502 2486 0.00742913 0 - 0.194117 2024-04-22T02:43:06.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2615 16 2502 2486 0.00742628 0 - 0.194117 2024-04-22T02:43:07.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2616 16 2502 2486 0.00742345 0 - 0.194117 2024-04-22T02:43:08.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2617 16 2502 2486 0.00742061 0 - 0.194117 2024-04-22T02:43:09.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2618 16 2502 2486 0.00741777 0 - 0.194117 2024-04-22T02:43:10.517 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2619 16 2502 2486 0.00741494 0 - 0.194117 2024-04-22T02:43:11.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:43:11.520399+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:43:11.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:43:11.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2620 16 2502 2486 0.00741211 0 - 0.194117 2024-04-22T02:43:12.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2621 16 2502 2486 0.00740928 0 - 0.194117 2024-04-22T02:43:13.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2622 16 2502 2486 0.00740646 0 - 0.194117 2024-04-22T02:43:14.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2623 16 2502 2486 0.00740364 0 - 0.194117 2024-04-22T02:43:15.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2624 16 2502 2486 0.00740081 0 - 0.194117 2024-04-22T02:43:16.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2625 16 2502 2486 0.00739799 0 - 0.194117 2024-04-22T02:43:17.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2626 16 2502 2486 0.00739518 0 - 0.194117 2024-04-22T02:43:18.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2627 16 2502 2486 0.00739236 0 - 0.194117 2024-04-22T02:43:19.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2628 16 2502 2486 0.00738955 0 - 0.194117 2024-04-22T02:43:20.518 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2629 16 2502 2486 0.00738674 0 - 0.194117 2024-04-22T02:43:21.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2630 16 2502 2486 0.00738393 0 - 0.194117 2024-04-22T02:43:22.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2631 16 2502 2486 0.00738112 0 - 0.194117 2024-04-22T02:43:23.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2632 16 2502 2486 0.00737832 0 - 0.194117 2024-04-22T02:43:24.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2633 16 2502 2486 0.00737552 0 - 0.194117 2024-04-22T02:43:25.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2634 16 2502 2486 0.00737272 0 - 0.194117 2024-04-22T02:43:26.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2635 16 2502 2486 0.00736992 0 - 0.194117 2024-04-22T02:43:27.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2636 16 2502 2486 0.00736712 0 - 0.194117 2024-04-22T02:43:28.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2637 16 2502 2486 0.00736433 0 - 0.194117 2024-04-22T02:43:29.519 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2638 16 2502 2486 0.00736154 0 - 0.194117 2024-04-22T02:43:30.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2639 16 2502 2486 0.00735875 0 - 0.194117 2024-04-22T02:43:31.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:43:31.522361+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:43:31.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:43:31.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2640 16 2502 2486 0.00735596 0 - 0.194117 2024-04-22T02:43:32.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2641 16 2502 2486 0.00735318 0 - 0.194117 2024-04-22T02:43:33.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2642 16 2502 2486 0.00735039 0 - 0.194117 2024-04-22T02:43:34.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2643 16 2502 2486 0.00734761 0 - 0.194117 2024-04-22T02:43:35.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2644 16 2502 2486 0.00734483 0 - 0.194117 2024-04-22T02:43:36.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2645 16 2502 2486 0.00734206 0 - 0.194117 2024-04-22T02:43:37.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2646 16 2502 2486 0.00733928 0 - 0.194117 2024-04-22T02:43:38.520 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2647 16 2502 2486 0.00733651 0 - 0.194117 2024-04-22T02:43:39.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2648 16 2502 2486 0.00733374 0 - 0.194117 2024-04-22T02:43:40.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2649 16 2502 2486 0.00733097 0 - 0.194117 2024-04-22T02:43:41.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2650 16 2502 2486 0.0073282 0 - 0.194117 2024-04-22T02:43:42.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2651 16 2502 2486 0.00732544 0 - 0.194117 2024-04-22T02:43:43.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2652 16 2502 2486 0.00732268 0 - 0.194117 2024-04-22T02:43:44.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2653 16 2502 2486 0.00731992 0 - 0.194117 2024-04-22T02:43:45.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2654 16 2502 2486 0.00731716 0 - 0.194117 2024-04-22T02:43:46.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2655 16 2502 2486 0.0073144 0 - 0.194117 2024-04-22T02:43:47.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2656 16 2502 2486 0.00731165 0 - 0.194117 2024-04-22T02:43:48.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2657 16 2502 2486 0.0073089 0 - 0.194117 2024-04-22T02:43:49.521 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2658 16 2502 2486 0.00730615 0 - 0.194117 2024-04-22T02:43:50.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2659 16 2502 2486 0.0073034 0 - 0.194117 2024-04-22T02:43:51.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:43:51.524533+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:43:51.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:43:51.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2660 16 2502 2486 0.00730065 0 - 0.194117 2024-04-22T02:43:52.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2661 16 2502 2486 0.00729791 0 - 0.194117 2024-04-22T02:43:53.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2662 16 2502 2486 0.00729517 0 - 0.194117 2024-04-22T02:43:54.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2663 16 2502 2486 0.00729243 0 - 0.194117 2024-04-22T02:43:55.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2664 16 2502 2486 0.00728969 0 - 0.194117 2024-04-22T02:43:56.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2665 16 2502 2486 0.00728696 0 - 0.194117 2024-04-22T02:43:57.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2666 16 2502 2486 0.00728422 0 - 0.194117 2024-04-22T02:43:58.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2667 16 2502 2486 0.00728149 0 - 0.194117 2024-04-22T02:43:59.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2668 16 2502 2486 0.00727876 0 - 0.194117 2024-04-22T02:44:00.522 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2669 16 2502 2486 0.00727604 0 - 0.194117 2024-04-22T02:44:01.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2670 16 2502 2486 0.00727331 0 - 0.194117 2024-04-22T02:44:02.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2671 16 2502 2486 0.00727059 0 - 0.194117 2024-04-22T02:44:03.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2672 16 2502 2486 0.00726787 0 - 0.194117 2024-04-22T02:44:04.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2673 16 2502 2486 0.00726515 0 - 0.194117 2024-04-22T02:44:05.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2674 16 2502 2486 0.00726243 0 - 0.194117 2024-04-22T02:44:06.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2675 16 2502 2486 0.00725972 0 - 0.194117 2024-04-22T02:44:07.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2676 16 2502 2486 0.007257 0 - 0.194117 2024-04-22T02:44:08.523 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2677 16 2502 2486 0.00725429 0 - 0.194117 2024-04-22T02:44:09.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2678 16 2502 2486 0.00725158 0 - 0.194117 2024-04-22T02:44:10.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2679 16 2502 2486 0.00724888 0 - 0.194117 2024-04-22T02:44:11.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:44:11.526679+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:44:11.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:44:11.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2680 16 2502 2486 0.00724617 0 - 0.194117 2024-04-22T02:44:12.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2681 16 2502 2486 0.00724347 0 - 0.194117 2024-04-22T02:44:13.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2682 16 2502 2486 0.00724077 0 - 0.194117 2024-04-22T02:44:14.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2683 16 2502 2486 0.00723807 0 - 0.194117 2024-04-22T02:44:15.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2684 16 2502 2486 0.00723537 0 - 0.194117 2024-04-22T02:44:16.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2685 16 2502 2486 0.00723268 0 - 0.194117 2024-04-22T02:44:17.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2686 16 2502 2486 0.00722998 0 - 0.194117 2024-04-22T02:44:18.524 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2687 16 2502 2486 0.00722729 0 - 0.194117 2024-04-22T02:44:19.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2688 16 2502 2486 0.00722461 0 - 0.194117 2024-04-22T02:44:20.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2689 16 2502 2486 0.00722192 0 - 0.194117 2024-04-22T02:44:21.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2690 16 2502 2486 0.00721923 0 - 0.194117 2024-04-22T02:44:22.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2691 16 2502 2486 0.00721655 0 - 0.194117 2024-04-22T02:44:23.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2692 16 2502 2486 0.00721387 0 - 0.194117 2024-04-22T02:44:24.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2693 16 2502 2486 0.00721119 0 - 0.194117 2024-04-22T02:44:25.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2694 16 2502 2486 0.00720851 0 - 0.194117 2024-04-22T02:44:26.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2695 16 2502 2486 0.00720584 0 - 0.194117 2024-04-22T02:44:27.525 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2696 16 2502 2486 0.00720317 0 - 0.194117 2024-04-22T02:44:28.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2697 16 2502 2486 0.0072005 0 - 0.194117 2024-04-22T02:44:29.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2698 16 2502 2486 0.00719783 0 - 0.194117 2024-04-22T02:44:30.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2699 16 2502 2486 0.00719516 0 - 0.194117 2024-04-22T02:44:31.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:44:31.528751+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:44:31.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:44:31.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2700 16 2502 2486 0.0071925 0 - 0.194117 2024-04-22T02:44:32.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2701 16 2502 2486 0.00718983 0 - 0.194117 2024-04-22T02:44:33.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2702 16 2502 2486 0.00718717 0 - 0.194117 2024-04-22T02:44:34.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2703 16 2502 2486 0.00718451 0 - 0.194117 2024-04-22T02:44:35.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2704 16 2502 2486 0.00718186 0 - 0.194117 2024-04-22T02:44:36.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2705 16 2502 2486 0.0071792 0 - 0.194117 2024-04-22T02:44:37.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2706 16 2502 2486 0.00717655 0 - 0.194117 2024-04-22T02:44:38.526 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2707 16 2502 2486 0.0071739 0 - 0.194117 2024-04-22T02:44:39.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2708 16 2502 2486 0.00717125 0 - 0.194117 2024-04-22T02:44:40.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2709 16 2502 2486 0.0071686 0 - 0.194117 2024-04-22T02:44:41.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2710 16 2502 2486 0.00716596 0 - 0.194117 2024-04-22T02:44:42.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2711 16 2502 2486 0.00716331 0 - 0.194117 2024-04-22T02:44:43.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2712 16 2502 2486 0.00716067 0 - 0.194117 2024-04-22T02:44:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:44:44.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:44:44.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2713 16 2502 2486 0.00715803 0 - 0.194117 2024-04-22T02:44:45.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2714 16 2502 2486 0.00715539 0 - 0.194117 2024-04-22T02:44:46.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:44:46.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:44:46.237+0000 7faafdced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:44:46.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2715 16 2502 2486 0.00715276 0 - 0.194117 2024-04-22T02:44:47.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2716 16 2502 2486 0.00715013 0 - 0.194117 2024-04-22T02:44:48.527 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2717 16 2502 2486 0.00714749 0 - 0.194117 2024-04-22T02:44:49.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2718 16 2502 2486 0.00714486 0 - 0.194117 2024-04-22T02:44:50.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2719 16 2502 2486 0.00714224 0 - 0.194117 2024-04-22T02:44:51.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:44:51.530605+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:44:51.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:44:51.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2720 16 2502 2486 0.00713961 0 - 0.194117 2024-04-22T02:44:52.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2721 16 2502 2486 0.00713699 0 - 0.194117 2024-04-22T02:44:53.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2722 16 2502 2486 0.00713436 0 - 0.194117 2024-04-22T02:44:54.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2723 16 2502 2486 0.00713174 0 - 0.194117 2024-04-22T02:44:55.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2724 16 2502 2486 0.00712913 0 - 0.194117 2024-04-22T02:44:56.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2725 16 2502 2486 0.00712651 0 - 0.194117 2024-04-22T02:44:57.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2726 16 2502 2486 0.0071239 0 - 0.194117 2024-04-22T02:44:58.528 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2727 16 2502 2486 0.00712128 0 - 0.194117 2024-04-22T02:44:59.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2728 16 2502 2486 0.00711867 0 - 0.194117 2024-04-22T02:45:00.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2729 16 2502 2486 0.00711606 0 - 0.194117 2024-04-22T02:45:01.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2730 16 2502 2486 0.00711346 0 - 0.194117 2024-04-22T02:45:02.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2731 16 2502 2486 0.00711085 0 - 0.194117 2024-04-22T02:45:03.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2732 16 2502 2486 0.00710825 0 - 0.194117 2024-04-22T02:45:04.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2733 16 2502 2486 0.00710565 0 - 0.194117 2024-04-22T02:45:05.529 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2734 16 2502 2486 0.00710305 0 - 0.194117 2024-04-22T02:45:06.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2735 16 2502 2486 0.00710045 0 - 0.194117 2024-04-22T02:45:07.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2736 16 2502 2486 0.00709786 0 - 0.194117 2024-04-22T02:45:08.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2737 16 2502 2486 0.00709526 0 - 0.194117 2024-04-22T02:45:09.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2738 16 2502 2486 0.00709267 0 - 0.194117 2024-04-22T02:45:10.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2739 16 2502 2486 0.00709008 0 - 0.194117 2024-04-22T02:45:11.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:45:11.532914+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:45:11.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:45:11.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2740 16 2502 2486 0.0070875 0 - 0.194117 2024-04-22T02:45:12.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2741 16 2502 2486 0.00708491 0 - 0.194117 2024-04-22T02:45:13.530 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2742 16 2502 2486 0.00708233 0 - 0.194117 2024-04-22T02:45:14.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2743 16 2502 2486 0.00707974 0 - 0.194117 2024-04-22T02:45:15.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2744 16 2502 2486 0.00707716 0 - 0.194117 2024-04-22T02:45:16.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2745 16 2502 2486 0.00707459 0 - 0.194117 2024-04-22T02:45:17.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2746 16 2502 2486 0.00707201 0 - 0.194117 2024-04-22T02:45:18.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2747 16 2502 2486 0.00706944 0 - 0.194117 2024-04-22T02:45:19.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2748 16 2502 2486 0.00706686 0 - 0.194117 2024-04-22T02:45:20.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2749 16 2502 2486 0.00706429 0 - 0.194117 2024-04-22T02:45:21.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2750 16 2502 2486 0.00706172 0 - 0.194117 2024-04-22T02:45:22.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2751 16 2502 2486 0.00705916 0 - 0.194117 2024-04-22T02:45:23.531 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2752 16 2502 2486 0.00705659 0 - 0.194117 2024-04-22T02:45:24.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2753 16 2502 2486 0.00705403 0 - 0.194117 2024-04-22T02:45:25.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2754 16 2502 2486 0.00705147 0 - 0.194117 2024-04-22T02:45:26.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2755 16 2502 2486 0.00704891 0 - 0.194117 2024-04-22T02:45:27.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2756 16 2502 2486 0.00704635 0 - 0.194117 2024-04-22T02:45:28.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2757 16 2502 2486 0.00704379 0 - 0.194117 2024-04-22T02:45:29.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2758 16 2502 2486 0.00704124 0 - 0.194117 2024-04-22T02:45:30.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2759 16 2502 2486 0.00703869 0 - 0.194117 2024-04-22T02:45:31.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:45:31.534975+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:45:31.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:45:31.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2760 16 2502 2486 0.00703614 0 - 0.194117 2024-04-22T02:45:32.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2761 16 2502 2486 0.00703359 0 - 0.194117 2024-04-22T02:45:33.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2762 16 2502 2486 0.00703104 0 - 0.194117 2024-04-22T02:45:34.532 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2763 16 2502 2486 0.0070285 0 - 0.194117 2024-04-22T02:45:35.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2764 16 2502 2486 0.00702596 0 - 0.194117 2024-04-22T02:45:36.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2765 16 2502 2486 0.00702341 0 - 0.194117 2024-04-22T02:45:37.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2766 16 2502 2486 0.00702088 0 - 0.194117 2024-04-22T02:45:38.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2767 16 2502 2486 0.00701834 0 - 0.194117 2024-04-22T02:45:39.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2768 16 2502 2486 0.0070158 0 - 0.194117 2024-04-22T02:45:40.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2769 16 2502 2486 0.00701327 0 - 0.194117 2024-04-22T02:45:41.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2770 16 2502 2486 0.00701074 0 - 0.194117 2024-04-22T02:45:42.533 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2771 16 2502 2486 0.00700821 0 - 0.194117 2024-04-22T02:45:43.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2772 16 2502 2486 0.00700568 0 - 0.194117 2024-04-22T02:45:44.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2773 16 2502 2486 0.00700315 0 - 0.194117 2024-04-22T02:45:45.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2774 16 2502 2486 0.00700063 0 - 0.194117 2024-04-22T02:45:46.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2775 16 2502 2486 0.0069981 0 - 0.194117 2024-04-22T02:45:47.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2776 16 2502 2486 0.00699558 0 - 0.194117 2024-04-22T02:45:48.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2777 16 2502 2486 0.00699306 0 - 0.194117 2024-04-22T02:45:49.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2778 16 2502 2486 0.00699055 0 - 0.194117 2024-04-22T02:45:50.534 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2779 16 2502 2486 0.00698803 0 - 0.194117 2024-04-22T02:45:51.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:45:51.537311+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:45:51.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:45:51.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2780 16 2502 2486 0.00698552 0 - 0.194117 2024-04-22T02:45:52.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2781 16 2502 2486 0.00698301 0 - 0.194117 2024-04-22T02:45:53.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2782 16 2502 2486 0.0069805 0 - 0.194117 2024-04-22T02:45:54.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2783 16 2502 2486 0.00697799 0 - 0.194117 2024-04-22T02:45:55.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2784 16 2502 2486 0.00697548 0 - 0.194117 2024-04-22T02:45:56.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2785 16 2502 2486 0.00697298 0 - 0.194117 2024-04-22T02:45:57.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2786 16 2502 2486 0.00697047 0 - 0.194117 2024-04-22T02:45:58.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2787 16 2502 2486 0.00696797 0 - 0.194117 2024-04-22T02:45:59.535 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2788 16 2502 2486 0.00696547 0 - 0.194117 2024-04-22T02:46:00.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2789 16 2502 2486 0.00696298 0 - 0.194117 2024-04-22T02:46:01.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2790 16 2502 2486 0.00696048 0 - 0.194117 2024-04-22T02:46:02.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2791 16 2502 2486 0.00695799 0 - 0.194117 2024-04-22T02:46:03.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2792 16 2502 2486 0.00695549 0 - 0.194117 2024-04-22T02:46:04.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2793 16 2502 2486 0.006953 0 - 0.194117 2024-04-22T02:46:05.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2794 16 2502 2486 0.00695051 0 - 0.194117 2024-04-22T02:46:06.536 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2795 16 2502 2486 0.00694803 0 - 0.194117 2024-04-22T02:46:07.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2796 16 2502 2486 0.00694554 0 - 0.194117 2024-04-22T02:46:08.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2797 16 2502 2486 0.00694306 0 - 0.194117 2024-04-22T02:46:09.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2798 16 2502 2486 0.00694058 0 - 0.194117 2024-04-22T02:46:10.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2799 16 2502 2486 0.0069381 0 - 0.194117 2024-04-22T02:46:11.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:46:11.539857+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:46:11.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:46:11.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2800 16 2502 2486 0.00693562 0 - 0.194117 2024-04-22T02:46:12.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2801 16 2502 2486 0.00693314 0 - 0.194117 2024-04-22T02:46:13.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2802 16 2502 2486 0.00693067 0 - 0.194117 2024-04-22T02:46:14.537 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2803 16 2502 2486 0.0069282 0 - 0.194117 2024-04-22T02:46:15.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2804 16 2502 2486 0.00692573 0 - 0.194117 2024-04-22T02:46:16.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2805 16 2502 2486 0.00692326 0 - 0.194117 2024-04-22T02:46:17.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2806 16 2502 2486 0.00692079 0 - 0.194117 2024-04-22T02:46:18.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2807 16 2502 2486 0.00691832 0 - 0.194117 2024-04-22T02:46:19.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2808 16 2502 2486 0.00691586 0 - 0.194117 2024-04-22T02:46:20.538 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2809 16 2502 2486 0.0069134 0 - 0.194117 2024-04-22T02:46:21.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2810 16 2502 2486 0.00691094 0 - 0.194117 2024-04-22T02:46:22.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2811 16 2502 2486 0.00690848 0 - 0.194117 2024-04-22T02:46:23.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2812 16 2502 2486 0.00690602 0 - 0.194117 2024-04-22T02:46:24.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2813 16 2502 2486 0.00690357 0 - 0.194117 2024-04-22T02:46:25.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2814 16 2502 2486 0.00690111 0 - 0.194117 2024-04-22T02:46:26.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2815 16 2502 2486 0.00689866 0 - 0.194117 2024-04-22T02:46:27.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2816 16 2502 2486 0.00689621 0 - 0.194117 2024-04-22T02:46:28.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2817 16 2502 2486 0.00689376 0 - 0.194117 2024-04-22T02:46:29.539 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2818 16 2502 2486 0.00689132 0 - 0.194117 2024-04-22T02:46:30.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2819 16 2502 2486 0.00688887 0 - 0.194117 2024-04-22T02:46:31.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:46:31.542577+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:46:31.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:46:31.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2820 16 2502 2486 0.00688643 0 - 0.194117 2024-04-22T02:46:32.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2821 16 2502 2486 0.00688399 0 - 0.194117 2024-04-22T02:46:33.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2822 16 2502 2486 0.00688155 0 - 0.194117 2024-04-22T02:46:34.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2823 16 2502 2486 0.00687911 0 - 0.194117 2024-04-22T02:46:35.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2824 16 2502 2486 0.00687668 0 - 0.194117 2024-04-22T02:46:36.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2825 16 2502 2486 0.00687424 0 - 0.194117 2024-04-22T02:46:37.540 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2826 16 2502 2486 0.00687181 0 - 0.194117 2024-04-22T02:46:38.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2827 16 2502 2486 0.00686938 0 - 0.194117 2024-04-22T02:46:39.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2828 16 2502 2486 0.00686695 0 - 0.194117 2024-04-22T02:46:40.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2829 16 2502 2486 0.00686452 0 - 0.194117 2024-04-22T02:46:41.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2830 16 2502 2486 0.0068621 0 - 0.194117 2024-04-22T02:46:42.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2831 16 2502 2486 0.00685967 0 - 0.194117 2024-04-22T02:46:43.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2832 16 2502 2486 0.00685725 0 - 0.194117 2024-04-22T02:46:44.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2833 16 2502 2486 0.00685483 0 - 0.194117 2024-04-22T02:46:45.541 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2834 16 2502 2486 0.00685241 0 - 0.194117 2024-04-22T02:46:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:46:46.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:46:46.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2835 16 2502 2486 0.00684999 0 - 0.194117 2024-04-22T02:46:47.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2836 16 2502 2486 0.00684758 0 - 0.194117 2024-04-22T02:46:48.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-22T02:46:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:46:48.274+0000 7f070a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:46:48.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2837 16 2502 2486 0.00684516 0 - 0.194117 2024-04-22T02:46:49.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2838 16 2502 2486 0.00684275 0 - 0.194117 2024-04-22T02:46:50.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2839 16 2502 2486 0.00684034 0 - 0.194117 2024-04-22T02:46:51.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:46:51.545118+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:46:51.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:46:51.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2840 16 2502 2486 0.00683793 0 - 0.194117 2024-04-22T02:46:52.542 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2841 16 2502 2486 0.00683553 0 - 0.194117 2024-04-22T02:46:53.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2842 16 2502 2486 0.00683312 0 - 0.194117 2024-04-22T02:46:54.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2843 16 2502 2486 0.00683072 0 - 0.194117 2024-04-22T02:46:55.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2844 16 2502 2486 0.00682832 0 - 0.194117 2024-04-22T02:46:56.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2845 16 2502 2486 0.00682592 0 - 0.194117 2024-04-22T02:46:57.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2846 16 2502 2486 0.00682352 0 - 0.194117 2024-04-22T02:46:58.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2847 16 2502 2486 0.00682112 0 - 0.194117 2024-04-22T02:46:59.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2848 16 2502 2486 0.00681873 0 - 0.194117 2024-04-22T02:47:00.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2849 16 2502 2486 0.00681633 0 - 0.194117 2024-04-22T02:47:01.543 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2850 16 2502 2486 0.00681394 0 - 0.194117 2024-04-22T02:47:02.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2851 16 2502 2486 0.00681155 0 - 0.194117 2024-04-22T02:47:03.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2852 16 2502 2486 0.00680916 0 - 0.194117 2024-04-22T02:47:04.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2853 16 2502 2486 0.00680678 0 - 0.194117 2024-04-22T02:47:05.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2854 16 2502 2486 0.00680439 0 - 0.194117 2024-04-22T02:47:06.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2855 16 2502 2486 0.00680201 0 - 0.194117 2024-04-22T02:47:07.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2856 16 2502 2486 0.00679963 0 - 0.194117 2024-04-22T02:47:08.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2857 16 2502 2486 0.00679725 0 - 0.194117 2024-04-22T02:47:09.544 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2858 16 2502 2486 0.00679487 0 - 0.194117 2024-04-22T02:47:10.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2859 16 2502 2486 0.00679249 0 - 0.194117 2024-04-22T02:47:11.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:47:11.547554+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:47:11.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:47:11.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2860 16 2502 2486 0.00679012 0 - 0.194117 2024-04-22T02:47:12.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2861 16 2502 2486 0.00678774 0 - 0.194117 2024-04-22T02:47:13.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2862 16 2502 2486 0.00678537 0 - 0.194117 2024-04-22T02:47:14.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2863 16 2502 2486 0.006783 0 - 0.194117 2024-04-22T02:47:15.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2864 16 2502 2486 0.00678063 0 - 0.194117 2024-04-22T02:47:16.545 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2865 16 2502 2486 0.00677827 0 - 0.194117 2024-04-22T02:47:17.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2866 16 2502 2486 0.0067759 0 - 0.194117 2024-04-22T02:47:18.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2867 16 2502 2486 0.00677354 0 - 0.194117 2024-04-22T02:47:19.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2868 16 2502 2486 0.00677117 0 - 0.194117 2024-04-22T02:47:20.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2869 16 2502 2486 0.00676881 0 - 0.194117 2024-04-22T02:47:21.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2870 16 2502 2486 0.00676646 0 - 0.194117 2024-04-22T02:47:22.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2871 16 2502 2486 0.0067641 0 - 0.194117 2024-04-22T02:47:23.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2872 16 2502 2486 0.00676174 0 - 0.194117 2024-04-22T02:47:24.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2873 16 2502 2486 0.00675939 0 - 0.194117 2024-04-22T02:47:25.546 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2874 16 2502 2486 0.00675704 0 - 0.194117 2024-04-22T02:47:26.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2875 16 2502 2486 0.00675469 0 - 0.194117 2024-04-22T02:47:27.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2876 16 2502 2486 0.00675234 0 - 0.194117 2024-04-22T02:47:28.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2877 16 2502 2486 0.00674999 0 - 0.194117 2024-04-22T02:47:29.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2878 16 2502 2486 0.00674765 0 - 0.194117 2024-04-22T02:47:30.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2879 16 2502 2486 0.0067453 0 - 0.194117 2024-04-22T02:47:31.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:47:31.550099+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:47:31.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:47:31.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2880 16 2502 2486 0.00674296 0 - 0.194117 2024-04-22T02:47:32.547 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2881 16 2502 2486 0.00674062 0 - 0.194117 2024-04-22T02:47:33.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2882 16 2502 2486 0.00673828 0 - 0.194117 2024-04-22T02:47:34.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2883 16 2502 2486 0.00673594 0 - 0.194117 2024-04-22T02:47:35.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2884 16 2502 2486 0.00673361 0 - 0.194117 2024-04-22T02:47:36.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2885 16 2502 2486 0.00673127 0 - 0.194117 2024-04-22T02:47:37.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2886 16 2502 2486 0.00672894 0 - 0.194117 2024-04-22T02:47:38.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2887 16 2502 2486 0.00672661 0 - 0.194117 2024-04-22T02:47:39.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2888 16 2502 2486 0.00672428 0 - 0.194117 2024-04-22T02:47:40.548 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2889 16 2502 2486 0.00672195 0 - 0.194117 2024-04-22T02:47:41.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2890 16 2502 2486 0.00671963 0 - 0.194117 2024-04-22T02:47:42.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2891 16 2502 2486 0.0067173 0 - 0.194117 2024-04-22T02:47:43.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2892 16 2502 2486 0.00671498 0 - 0.194117 2024-04-22T02:47:44.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2893 16 2502 2486 0.00671266 0 - 0.194117 2024-04-22T02:47:45.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2894 16 2502 2486 0.00671034 0 - 0.194117 2024-04-22T02:47:46.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2895 16 2502 2486 0.00670802 0 - 0.194117 2024-04-22T02:47:47.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2896 16 2502 2486 0.00670571 0 - 0.194117 2024-04-22T02:47:48.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2897 16 2502 2486 0.00670339 0 - 0.194117 2024-04-22T02:47:49.549 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2898 16 2502 2486 0.00670108 0 - 0.194117 2024-04-22T02:47:50.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2899 16 2502 2486 0.00669877 0 - 0.194117 2024-04-22T02:47:51.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:47:51.552543+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:47:51.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:47:51.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2900 16 2502 2486 0.00669646 0 - 0.194117 2024-04-22T02:47:52.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2901 16 2502 2486 0.00669415 0 - 0.194117 2024-04-22T02:47:53.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2902 16 2502 2486 0.00669184 0 - 0.194117 2024-04-22T02:47:54.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2903 16 2502 2486 0.00668954 0 - 0.194117 2024-04-22T02:47:55.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2904 16 2502 2486 0.00668723 0 - 0.194117 2024-04-22T02:47:56.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2905 16 2502 2486 0.00668493 0 - 0.194117 2024-04-22T02:47:57.550 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2906 16 2502 2486 0.00668263 0 - 0.194117 2024-04-22T02:47:58.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2907 16 2502 2486 0.00668033 0 - 0.194117 2024-04-22T02:47:59.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2908 16 2502 2486 0.00667803 0 - 0.194117 2024-04-22T02:48:00.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2909 16 2502 2486 0.00667574 0 - 0.194117 2024-04-22T02:48:01.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2910 16 2502 2486 0.00667345 0 - 0.194117 2024-04-22T02:48:02.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2911 16 2502 2486 0.00667115 0 - 0.194117 2024-04-22T02:48:03.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2912 16 2502 2486 0.00666886 0 - 0.194117 2024-04-22T02:48:04.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2913 16 2502 2486 0.00666657 0 - 0.194117 2024-04-22T02:48:05.551 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2914 16 2502 2486 0.00666428 0 - 0.194117 2024-04-22T02:48:06.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2915 16 2502 2486 0.006662 0 - 0.194117 2024-04-22T02:48:07.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2916 16 2502 2486 0.00665971 0 - 0.194117 2024-04-22T02:48:08.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2917 16 2502 2486 0.00665743 0 - 0.194117 2024-04-22T02:48:09.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2918 16 2502 2486 0.00665515 0 - 0.194117 2024-04-22T02:48:10.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2919 16 2502 2486 0.00665287 0 - 0.194117 2024-04-22T02:48:11.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:48:11.554995+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:48:11.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:48:11.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2920 16 2502 2486 0.00665059 0 - 0.194117 2024-04-22T02:48:12.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2921 16 2502 2486 0.00664831 0 - 0.194117 2024-04-22T02:48:13.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2922 16 2502 2486 0.00664604 0 - 0.194117 2024-04-22T02:48:14.552 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2923 16 2502 2486 0.00664376 0 - 0.194117 2024-04-22T02:48:15.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2924 16 2502 2486 0.00664149 0 - 0.194117 2024-04-22T02:48:16.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2925 16 2502 2486 0.00663922 0 - 0.194117 2024-04-22T02:48:17.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2926 16 2502 2486 0.00663695 0 - 0.194117 2024-04-22T02:48:18.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2927 16 2502 2486 0.00663469 0 - 0.194117 2024-04-22T02:48:19.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2928 16 2502 2486 0.00663242 0 - 0.194117 2024-04-22T02:48:20.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2929 16 2502 2486 0.00663015 0 - 0.194117 2024-04-22T02:48:21.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2930 16 2502 2486 0.00662789 0 - 0.194117 2024-04-22T02:48:22.553 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2931 16 2502 2486 0.00662563 0 - 0.194117 2024-04-22T02:48:23.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2932 16 2502 2486 0.00662337 0 - 0.194117 2024-04-22T02:48:24.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2933 16 2502 2486 0.00662111 0 - 0.194117 2024-04-22T02:48:25.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2934 16 2502 2486 0.00661886 0 - 0.194117 2024-04-22T02:48:26.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2935 16 2502 2486 0.0066166 0 - 0.194117 2024-04-22T02:48:27.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2936 16 2502 2486 0.00661435 0 - 0.194117 2024-04-22T02:48:28.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2937 16 2502 2486 0.0066121 0 - 0.194117 2024-04-22T02:48:29.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2938 16 2502 2486 0.00660984 0 - 0.194117 2024-04-22T02:48:30.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2939 16 2502 2486 0.0066076 0 - 0.194117 2024-04-22T02:48:31.554 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:48:31.557294+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:48:31.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:48:31.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2940 16 2502 2486 0.00660535 0 - 0.194117 2024-04-22T02:48:32.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2941 16 2502 2486 0.0066031 0 - 0.194117 2024-04-22T02:48:33.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2942 16 2502 2486 0.00660086 0 - 0.194117 2024-04-22T02:48:34.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2943 16 2502 2486 0.00659861 0 - 0.194117 2024-04-22T02:48:35.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2944 16 2502 2486 0.00659637 0 - 0.194117 2024-04-22T02:48:36.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2945 16 2502 2486 0.00659413 0 - 0.194117 2024-04-22T02:48:37.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2946 16 2502 2486 0.0065919 0 - 0.194117 2024-04-22T02:48:38.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2947 16 2502 2486 0.00658966 0 - 0.194117 2024-04-22T02:48:39.555 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2948 16 2502 2486 0.00658742 0 - 0.194117 2024-04-22T02:48:40.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2949 16 2502 2486 0.00658519 0 - 0.194117 2024-04-22T02:48:41.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2950 16 2502 2486 0.00658296 0 - 0.194117 2024-04-22T02:48:42.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2951 16 2502 2486 0.00658073 0 - 0.194117 2024-04-22T02:48:43.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2952 16 2502 2486 0.0065785 0 - 0.194117 2024-04-22T02:48:44.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2953 16 2502 2486 0.00657627 0 - 0.194117 2024-04-22T02:48:45.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2954 16 2502 2486 0.00657404 0 - 0.194117 2024-04-22T02:48:46.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2955 16 2502 2486 0.00657182 0 - 0.194117 2024-04-22T02:48:47.556 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2956 16 2502 2486 0.00656959 0 - 0.194117 2024-04-22T02:48:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:48:48.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:48:48.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2957 16 2502 2486 0.00656737 0 - 0.194117 2024-04-22T02:48:49.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2958 16 2502 2486 0.00656515 0 - 0.194117 2024-04-22T02:48:50.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-22T02:48:50.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:48:50.307+0000 7fcd960ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:48:50.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2959 16 2502 2486 0.00656293 0 - 0.194117 2024-04-22T02:48:51.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:48:51.559792+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:48:51.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:48:51.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2960 16 2502 2486 0.00656072 0 - 0.194117 2024-04-22T02:48:52.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2961 16 2502 2486 0.0065585 0 - 0.194117 2024-04-22T02:48:53.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2962 16 2502 2486 0.00655629 0 - 0.194117 2024-04-22T02:48:54.557 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2963 16 2502 2486 0.00655407 0 - 0.194117 2024-04-22T02:48:55.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2964 16 2502 2486 0.00655186 0 - 0.194117 2024-04-22T02:48:56.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2965 16 2502 2486 0.00654965 0 - 0.194117 2024-04-22T02:48:57.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2966 16 2502 2486 0.00654744 0 - 0.194117 2024-04-22T02:48:58.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2967 16 2502 2486 0.00654524 0 - 0.194117 2024-04-22T02:48:59.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2968 16 2502 2486 0.00654303 0 - 0.194117 2024-04-22T02:49:00.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2969 16 2502 2486 0.00654083 0 - 0.194117 2024-04-22T02:49:01.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2970 16 2502 2486 0.00653863 0 - 0.194117 2024-04-22T02:49:02.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2971 16 2502 2486 0.00653643 0 - 0.194117 2024-04-22T02:49:03.558 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2972 16 2502 2486 0.00653423 0 - 0.194117 2024-04-22T02:49:04.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2973 16 2502 2486 0.00653203 0 - 0.194117 2024-04-22T02:49:05.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2974 16 2502 2486 0.00652983 0 - 0.194117 2024-04-22T02:49:06.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2975 16 2502 2486 0.00652764 0 - 0.194117 2024-04-22T02:49:07.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2976 16 2502 2486 0.00652544 0 - 0.194117 2024-04-22T02:49:08.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2977 16 2502 2486 0.00652325 0 - 0.194117 2024-04-22T02:49:09.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2978 16 2502 2486 0.00652106 0 - 0.194117 2024-04-22T02:49:10.559 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2979 16 2502 2486 0.00651887 0 - 0.194117 2024-04-22T02:49:11.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:49:11.562277+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:49:11.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:49:11.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2980 16 2502 2486 0.00651668 0 - 0.194117 2024-04-22T02:49:12.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2981 16 2502 2486 0.0065145 0 - 0.194117 2024-04-22T02:49:13.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2982 16 2502 2486 0.00651231 0 - 0.194117 2024-04-22T02:49:14.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2983 16 2502 2486 0.00651013 0 - 0.194117 2024-04-22T02:49:15.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2984 16 2502 2486 0.00650795 0 - 0.194117 2024-04-22T02:49:16.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2985 16 2502 2486 0.00650577 0 - 0.194117 2024-04-22T02:49:17.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2986 16 2502 2486 0.00650359 0 - 0.194117 2024-04-22T02:49:18.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2987 16 2502 2486 0.00650141 0 - 0.194117 2024-04-22T02:49:19.560 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2988 16 2502 2486 0.00649924 0 - 0.194117 2024-04-22T02:49:20.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2989 16 2502 2486 0.00649706 0 - 0.194117 2024-04-22T02:49:21.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2990 16 2502 2486 0.00649489 0 - 0.194117 2024-04-22T02:49:22.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2991 16 2502 2486 0.00649272 0 - 0.194117 2024-04-22T02:49:23.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2992 16 2502 2486 0.00649055 0 - 0.194117 2024-04-22T02:49:24.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2993 16 2502 2486 0.00648838 0 - 0.194117 2024-04-22T02:49:25.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2994 16 2502 2486 0.00648621 0 - 0.194117 2024-04-22T02:49:26.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2995 16 2502 2486 0.00648405 0 - 0.194117 2024-04-22T02:49:27.561 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2996 16 2502 2486 0.00648188 0 - 0.194117 2024-04-22T02:49:28.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2997 16 2502 2486 0.00647972 0 - 0.194117 2024-04-22T02:49:29.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2998 16 2502 2486 0.00647756 0 - 0.194117 2024-04-22T02:49:30.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2999 16 2502 2486 0.0064754 0 - 0.194117 2024-04-22T02:49:31.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:49:31.564730+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:49:31.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:49:31.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3000 16 2502 2486 0.00647324 0 - 0.194117 2024-04-22T02:49:32.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3001 16 2502 2486 0.00647108 0 - 0.194117 2024-04-22T02:49:33.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3002 16 2502 2486 0.00646893 0 - 0.194117 2024-04-22T02:49:34.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3003 16 2502 2486 0.00646677 0 - 0.194117 2024-04-22T02:49:35.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3004 16 2502 2486 0.00646462 0 - 0.194117 2024-04-22T02:49:36.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3005 16 2502 2486 0.00646247 0 - 0.194117 2024-04-22T02:49:37.562 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3006 16 2502 2486 0.00646032 0 - 0.194117 2024-04-22T02:49:38.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3007 16 2502 2486 0.00645817 0 - 0.194117 2024-04-22T02:49:39.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3008 16 2502 2486 0.00645602 0 - 0.194117 2024-04-22T02:49:40.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3009 16 2502 2486 0.00645388 0 - 0.194117 2024-04-22T02:49:41.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3010 16 2502 2486 0.00645173 0 - 0.194117 2024-04-22T02:49:42.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3011 16 2502 2486 0.00644959 0 - 0.194117 2024-04-22T02:49:43.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3012 16 2502 2486 0.00644745 0 - 0.194117 2024-04-22T02:49:44.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3013 16 2502 2486 0.00644531 0 - 0.194117 2024-04-22T02:49:45.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3014 16 2502 2486 0.00644317 0 - 0.194117 2024-04-22T02:49:46.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3015 16 2502 2486 0.00644104 0 - 0.194117 2024-04-22T02:49:47.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3016 16 2502 2486 0.0064389 0 - 0.194117 2024-04-22T02:49:48.563 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3017 16 2502 2486 0.00643677 0 - 0.194117 2024-04-22T02:49:49.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3018 16 2502 2486 0.00643463 0 - 0.194117 2024-04-22T02:49:50.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3019 16 2502 2486 0.0064325 0 - 0.194117 2024-04-22T02:49:51.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:49:51.566547+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:49:51.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:49:51.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3020 16 2502 2486 0.00643037 0 - 0.194117 2024-04-22T02:49:52.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3021 16 2502 2486 0.00642824 0 - 0.194117 2024-04-22T02:49:53.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3022 16 2502 2486 0.00642612 0 - 0.194117 2024-04-22T02:49:54.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3023 16 2502 2486 0.00642399 0 - 0.194117 2024-04-22T02:49:55.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3024 16 2502 2486 0.00642187 0 - 0.194117 2024-04-22T02:49:56.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3025 16 2502 2486 0.00641974 0 - 0.194117 2024-04-22T02:49:57.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3026 16 2502 2486 0.00641762 0 - 0.194117 2024-04-22T02:49:58.564 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3027 16 2502 2486 0.0064155 0 - 0.194117 2024-04-22T02:49:59.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3028 16 2502 2486 0.00641338 0 - 0.194117 2024-04-22T02:50:00.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3029 16 2502 2486 0.00641127 0 - 0.194117 2024-04-22T02:50:01.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3030 16 2502 2486 0.00640915 0 - 0.194117 2024-04-22T02:50:02.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3031 16 2502 2486 0.00640703 0 - 0.194117 2024-04-22T02:50:03.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3032 16 2502 2486 0.00640492 0 - 0.194117 2024-04-22T02:50:04.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3033 16 2502 2486 0.00640281 0 - 0.194117 2024-04-22T02:50:05.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3034 16 2502 2486 0.0064007 0 - 0.194117 2024-04-22T02:50:06.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3035 16 2502 2486 0.00639859 0 - 0.194117 2024-04-22T02:50:07.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3036 16 2502 2486 0.00639648 0 - 0.194117 2024-04-22T02:50:08.565 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3037 16 2502 2486 0.00639438 0 - 0.194117 2024-04-22T02:50:09.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3038 16 2502 2486 0.00639227 0 - 0.194117 2024-04-22T02:50:10.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3039 16 2502 2486 0.00639017 0 - 0.194117 2024-04-22T02:50:11.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:50:11.568604+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:50:11.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:50:11.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3040 16 2502 2486 0.00638807 0 - 0.194117 2024-04-22T02:50:12.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3041 16 2502 2486 0.00638597 0 - 0.194117 2024-04-22T02:50:13.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3042 16 2502 2486 0.00638387 0 - 0.194117 2024-04-22T02:50:14.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3043 16 2502 2486 0.00638177 0 - 0.194117 2024-04-22T02:50:15.566 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3044 16 2502 2486 0.00637967 0 - 0.194117 2024-04-22T02:50:15.879 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-22T02:50:16.107 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2bcda724b5df4ad394b7676fe8d228e9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-04-22T02:50:16.109 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-22T02:50:16.124 INFO:tasks.thrashosds:joining thrashosds 2024-04-22T02:50:16.124 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-22T02:50:16.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3045 16 2502 2486 0.00637758 0 - 0.194117 2024-04-22T02:50:17.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3046 16 2502 2486 0.00637548 0 - 0.194117 2024-04-22T02:50:18.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3047 16 2502 2486 0.00637339 0 - 0.194117 2024-04-22T02:50:19.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3048 16 2502 2486 0.0063713 0 - 0.194117 2024-04-22T02:50:20.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3049 16 2502 2486 0.00636921 0 - 0.194117 2024-04-22T02:50:21.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3050 16 2502 2486 0.00636712 0 - 0.194117 2024-04-22T02:50:22.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3051 16 2502 2486 0.00636503 0 - 0.194117 2024-04-22T02:50:23.567 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3052 16 2502 2486 0.00636295 0 - 0.194117 2024-04-22T02:50:24.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3053 16 2502 2486 0.00636087 0 - 0.194117 2024-04-22T02:50:25.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3054 16 2502 2486 0.00635878 0 - 0.194117 2024-04-22T02:50:26.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3055 16 2502 2486 0.0063567 0 - 0.194117 2024-04-22T02:50:27.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3056 16 2502 2486 0.00635462 0 - 0.194117 2024-04-22T02:50:28.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3057 16 2502 2486 0.00635254 0 - 0.194117 2024-04-22T02:50:29.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3058 16 2502 2486 0.00635046 0 - 0.194117 2024-04-22T02:50:30.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3059 16 2502 2486 0.00634839 0 - 0.194117 2024-04-22T02:50:31.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:50:31.571084+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:50:31.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:50:31.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3060 16 2502 2486 0.00634631 0 - 0.194117 2024-04-22T02:50:32.568 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3061 16 2502 2486 0.00634424 0 - 0.194117 2024-04-22T02:50:33.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3062 16 2502 2486 0.00634217 0 - 0.194117 2024-04-22T02:50:34.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3063 16 2502 2486 0.0063401 0 - 0.194117 2024-04-22T02:50:35.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3064 16 2502 2486 0.00633803 0 - 0.194117 2024-04-22T02:50:36.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3065 16 2502 2486 0.00633596 0 - 0.194117 2024-04-22T02:50:37.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3066 16 2502 2486 0.00633389 0 - 0.194117 2024-04-22T02:50:38.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3067 16 2502 2486 0.00633183 0 - 0.194117 2024-04-22T02:50:39.569 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3068 16 2502 2486 0.00632977 0 - 0.194117 2024-04-22T02:50:40.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3069 16 2502 2486 0.0063277 0 - 0.194117 2024-04-22T02:50:41.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3070 16 2502 2486 0.00632564 0 - 0.194117 2024-04-22T02:50:42.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3071 16 2502 2486 0.00632358 0 - 0.194117 2024-04-22T02:50:43.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3072 16 2502 2486 0.00632152 0 - 0.194117 2024-04-22T02:50:44.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3073 16 2502 2486 0.00631947 0 - 0.194117 2024-04-22T02:50:45.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3074 16 2502 2486 0.00631741 0 - 0.194117 2024-04-22T02:50:46.570 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3075 16 2502 2486 0.00631536 0 - 0.194117 2024-04-22T02:50:47.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3076 16 2502 2486 0.0063133 0 - 0.194117 2024-04-22T02:50:48.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3077 16 2502 2486 0.00631125 0 - 0.194117 2024-04-22T02:50:49.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3078 16 2502 2486 0.0063092 0 - 0.194117 2024-04-22T02:50:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:50:50.226 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-22T02:50:50.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3079 16 2502 2486 0.00630715 0 - 0.194117 2024-04-22T02:50:51.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:50:51.573774+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:50:51.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:50:51.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3080 16 2502 2486 0.0063051 0 - 0.194117 2024-04-22T02:50:52.227 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T02:50:52.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T02:50:52.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:50:52.344+0000 7f526c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:50:52.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3081 16 2502 2486 0.00630306 0 - 0.194117 2024-04-22T02:50:53.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3082 16 2502 2486 0.00630101 0 - 0.194117 2024-04-22T02:50:54.571 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3083 16 2502 2486 0.00629897 0 - 0.194117 2024-04-22T02:50:55.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3084 16 2502 2486 0.00629693 0 - 0.194117 2024-04-22T02:50:56.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3085 16 2502 2486 0.00629488 0 - 0.194117 2024-04-22T02:50:57.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3086 16 2502 2486 0.00629284 0 - 0.194117 2024-04-22T02:50:58.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3087 16 2502 2486 0.00629081 0 - 0.194117 2024-04-22T02:50:59.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3088 16 2502 2486 0.00628877 0 - 0.194117 2024-04-22T02:51:00.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3089 16 2502 2486 0.00628673 0 - 0.194117 2024-04-22T02:51:01.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3090 16 2502 2486 0.0062847 0 - 0.194117 2024-04-22T02:51:02.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3091 16 2502 2486 0.00628266 0 - 0.194117 2024-04-22T02:51:03.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3092 16 2502 2486 0.00628063 0 - 0.194117 2024-04-22T02:51:04.572 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3093 16 2502 2486 0.0062786 0 - 0.194117 2024-04-22T02:51:05.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3094 16 2502 2486 0.00627657 0 - 0.194117 2024-04-22T02:51:06.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3095 16 2502 2486 0.00627455 0 - 0.194117 2024-04-22T02:51:07.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3096 16 2502 2486 0.00627252 0 - 0.194117 2024-04-22T02:51:08.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3097 16 2502 2486 0.00627049 0 - 0.194117 2024-04-22T02:51:09.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3098 16 2502 2486 0.00626847 0 - 0.194117 2024-04-22T02:51:10.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3099 16 2502 2486 0.00626645 0 - 0.194117 2024-04-22T02:51:11.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:51:11.576054+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:51:11.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:51:11.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3100 16 2502 2486 0.00626442 0 - 0.194117 2024-04-22T02:51:12.573 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3101 16 2502 2486 0.0062624 0 - 0.194117 2024-04-22T02:51:13.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3102 16 2502 2486 0.00626039 0 - 0.194117 2024-04-22T02:51:14.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3103 16 2502 2486 0.00625837 0 - 0.194117 2024-04-22T02:51:15.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3104 16 2502 2486 0.00625635 0 - 0.194117 2024-04-22T02:51:16.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3105 16 2502 2486 0.00625434 0 - 0.194117 2024-04-22T02:51:17.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3106 16 2502 2486 0.00625232 0 - 0.194117 2024-04-22T02:51:18.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3107 16 2502 2486 0.00625031 0 - 0.194117 2024-04-22T02:51:19.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3108 16 2502 2486 0.0062483 0 - 0.194117 2024-04-22T02:51:20.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3109 16 2502 2486 0.00624629 0 - 0.194117 2024-04-22T02:51:21.574 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3110 16 2502 2486 0.00624428 0 - 0.194117 2024-04-22T02:51:22.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3111 16 2502 2486 0.00624227 0 - 0.194117 2024-04-22T02:51:23.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3112 16 2502 2486 0.00624027 0 - 0.194117 2024-04-22T02:51:24.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3113 16 2502 2486 0.00623826 0 - 0.194117 2024-04-22T02:51:25.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3114 16 2502 2486 0.00623626 0 - 0.194117 2024-04-22T02:51:26.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3115 16 2502 2486 0.00623426 0 - 0.194117 2024-04-22T02:51:27.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3116 16 2502 2486 0.00623226 0 - 0.194117 2024-04-22T02:51:28.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3117 16 2502 2486 0.00623026 0 - 0.194117 2024-04-22T02:51:29.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3118 16 2502 2486 0.00622826 0 - 0.194117 2024-04-22T02:51:30.575 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3119 16 2502 2486 0.00622626 0 - 0.194117 2024-04-22T02:51:31.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:51:31.578381+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:51:31.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:51:31.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3120 16 2502 2486 0.00622427 0 - 0.194117 2024-04-22T02:51:32.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3121 16 2502 2486 0.00622227 0 - 0.194117 2024-04-22T02:51:33.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3122 16 2502 2486 0.00622028 0 - 0.194117 2024-04-22T02:51:34.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3123 16 2502 2486 0.00621829 0 - 0.194117 2024-04-22T02:51:35.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3124 16 2502 2486 0.0062163 0 - 0.194117 2024-04-22T02:51:36.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3125 16 2502 2486 0.00621431 0 - 0.194117 2024-04-22T02:51:37.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3126 16 2502 2486 0.00621232 0 - 0.194117 2024-04-22T02:51:38.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3127 16 2502 2486 0.00621033 0 - 0.194117 2024-04-22T02:51:39.576 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3128 16 2502 2486 0.00620835 0 - 0.194117 2024-04-22T02:51:40.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3129 16 2502 2486 0.00620637 0 - 0.194117 2024-04-22T02:51:41.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3130 16 2502 2486 0.00620438 0 - 0.194117 2024-04-22T02:51:42.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3131 16 2502 2486 0.0062024 0 - 0.194117 2024-04-22T02:51:43.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3132 16 2502 2486 0.00620042 0 - 0.194117 2024-04-22T02:51:44.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3133 16 2502 2486 0.00619844 0 - 0.194117 2024-04-22T02:51:45.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3134 16 2502 2486 0.00619646 0 - 0.194117 2024-04-22T02:51:46.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3135 16 2502 2486 0.00619449 0 - 0.194117 2024-04-22T02:51:47.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3136 16 2502 2486 0.00619251 0 - 0.194117 2024-04-22T02:51:48.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3137 16 2502 2486 0.00619054 0 - 0.194117 2024-04-22T02:51:49.577 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3138 16 2502 2486 0.00618857 0 - 0.194117 2024-04-22T02:51:50.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3139 16 2502 2486 0.00618659 0 - 0.194117 2024-04-22T02:51:51.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:51:51.580422+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:51:51.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:51:51.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3140 16 2502 2486 0.00618462 0 - 0.194117 2024-04-22T02:51:52.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3141 16 2502 2486 0.00618265 0 - 0.194117 2024-04-22T02:51:53.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3142 16 2502 2486 0.00618069 0 - 0.194117 2024-04-22T02:51:54.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3143 16 2502 2486 0.00617872 0 - 0.194117 2024-04-22T02:51:55.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3144 16 2502 2486 0.00617676 0 - 0.194117 2024-04-22T02:51:56.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3145 16 2502 2486 0.00617479 0 - 0.194117 2024-04-22T02:51:57.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3146 16 2502 2486 0.00617283 0 - 0.194117 2024-04-22T02:51:58.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3147 16 2502 2486 0.00617087 0 - 0.194117 2024-04-22T02:51:59.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3148 16 2502 2486 0.00616891 0 - 0.194117 2024-04-22T02:52:00.578 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3149 16 2502 2486 0.00616695 0 - 0.194117 2024-04-22T02:52:01.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3150 16 2502 2486 0.00616499 0 - 0.194117 2024-04-22T02:52:02.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3151 16 2502 2486 0.00616303 0 - 0.194117 2024-04-22T02:52:03.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3152 16 2502 2486 0.00616108 0 - 0.194117 2024-04-22T02:52:04.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3153 16 2502 2486 0.00615912 0 - 0.194117 2024-04-22T02:52:05.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3154 16 2502 2486 0.00615717 0 - 0.194117 2024-04-22T02:52:06.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3155 16 2502 2486 0.00615522 0 - 0.194117 2024-04-22T02:52:07.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3156 16 2502 2486 0.00615327 0 - 0.194117 2024-04-22T02:52:08.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3157 16 2502 2486 0.00615132 0 - 0.194117 2024-04-22T02:52:09.579 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3158 16 2502 2486 0.00614937 0 - 0.194117 2024-04-22T02:52:10.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3159 16 2502 2486 0.00614743 0 - 0.194117 2024-04-22T02:52:11.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:52:11.582452+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:52:11.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:52:11.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3160 16 2502 2486 0.00614548 0 - 0.194117 2024-04-22T02:52:12.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3161 16 2502 2486 0.00614354 0 - 0.194117 2024-04-22T02:52:13.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3162 16 2502 2486 0.00614159 0 - 0.194117 2024-04-22T02:52:14.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3163 16 2502 2486 0.00613965 0 - 0.194117 2024-04-22T02:52:15.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3164 16 2502 2486 0.00613771 0 - 0.194117 2024-04-22T02:52:16.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3165 16 2502 2486 0.00613577 0 - 0.194117 2024-04-22T02:52:17.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3166 16 2502 2486 0.00613383 0 - 0.194117 2024-04-22T02:52:18.580 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3167 16 2502 2486 0.0061319 0 - 0.194117 2024-04-22T02:52:19.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3168 16 2502 2486 0.00612996 0 - 0.194117 2024-04-22T02:52:20.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3169 16 2502 2486 0.00612803 0 - 0.194117 2024-04-22T02:52:21.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3170 16 2502 2486 0.00612609 0 - 0.194117 2024-04-22T02:52:22.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3171 16 2502 2486 0.00612416 0 - 0.194117 2024-04-22T02:52:23.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3172 16 2502 2486 0.00612223 0 - 0.194117 2024-04-22T02:52:24.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3173 16 2502 2486 0.0061203 0 - 0.194117 2024-04-22T02:52:25.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3174 16 2502 2486 0.00611837 0 - 0.194117 2024-04-22T02:52:26.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3175 16 2502 2486 0.00611645 0 - 0.194117 2024-04-22T02:52:27.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3176 16 2502 2486 0.00611452 0 - 0.194117 2024-04-22T02:52:28.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3177 16 2502 2486 0.0061126 0 - 0.194117 2024-04-22T02:52:29.581 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3178 16 2502 2486 0.00611067 0 - 0.194117 2024-04-22T02:52:30.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3179 16 2502 2486 0.00610875 0 - 0.194117 2024-04-22T02:52:31.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:52:31.584478+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:52:31.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:52:31.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3180 16 2502 2486 0.00610683 0 - 0.194117 2024-04-22T02:52:32.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3181 16 2502 2486 0.00610491 0 - 0.194117 2024-04-22T02:52:33.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3182 16 2502 2486 0.00610299 0 - 0.194117 2024-04-22T02:52:34.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3183 16 2502 2486 0.00610107 0 - 0.194117 2024-04-22T02:52:35.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3184 16 2502 2486 0.00609916 0 - 0.194117 2024-04-22T02:52:36.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3185 16 2502 2486 0.00609724 0 - 0.194117 2024-04-22T02:52:37.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3186 16 2502 2486 0.00609533 0 - 0.194117 2024-04-22T02:52:38.582 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3187 16 2502 2486 0.00609342 0 - 0.194117 2024-04-22T02:52:39.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3188 16 2502 2486 0.00609151 0 - 0.194117 2024-04-22T02:52:40.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3189 16 2502 2486 0.0060896 0 - 0.194117 2024-04-22T02:52:41.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3190 16 2502 2486 0.00608769 0 - 0.194117 2024-04-22T02:52:42.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3191 16 2502 2486 0.00608578 0 - 0.194117 2024-04-22T02:52:43.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3192 16 2502 2486 0.00608387 0 - 0.194117 2024-04-22T02:52:44.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3193 16 2502 2486 0.00608197 0 - 0.194117 2024-04-22T02:52:45.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3194 16 2502 2486 0.00608006 0 - 0.194117 2024-04-22T02:52:46.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3195 16 2502 2486 0.00607816 0 - 0.194117 2024-04-22T02:52:47.583 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3196 16 2502 2486 0.00607626 0 - 0.194117 2024-04-22T02:52:48.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3197 16 2502 2486 0.00607436 0 - 0.194117 2024-04-22T02:52:49.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3198 16 2502 2486 0.00607246 0 - 0.194117 2024-04-22T02:52:50.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3199 16 2502 2486 0.00607056 0 - 0.194117 2024-04-22T02:52:51.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:52:51.586609+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:52:51.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:52:51.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3200 16 2502 2486 0.00606866 0 - 0.194117 2024-04-22T02:52:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:52:52.263 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-22T02:52:52.275 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T02:52:52.284 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T02:52:52.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T02:52:52.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-04-22T02:52:52.405+0000 7f817f8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T02:52:52.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3201 16 2502 2486 0.00606677 0 - 0.194117 2024-04-22T02:52:53.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3202 16 2502 2486 0.00606487 0 - 0.194117 2024-04-22T02:52:54.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3203 16 2502 2486 0.00606298 0 - 0.194117 2024-04-22T02:52:55.584 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3204 16 2502 2486 0.00606109 0 - 0.194117 2024-04-22T02:52:56.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3205 16 2502 2486 0.00605919 0 - 0.194117 2024-04-22T02:52:57.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3206 16 2502 2486 0.0060573 0 - 0.194117 2024-04-22T02:52:58.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3207 16 2502 2486 0.00605542 0 - 0.194117 2024-04-22T02:52:59.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3208 16 2502 2486 0.00605353 0 - 0.194117 2024-04-22T02:53:00.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3209 16 2502 2486 0.00605164 0 - 0.194117 2024-04-22T02:53:01.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3210 16 2502 2486 0.00604976 0 - 0.194117 2024-04-22T02:53:02.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3211 16 2502 2486 0.00604787 0 - 0.194117 2024-04-22T02:53:03.585 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3212 16 2502 2486 0.00604599 0 - 0.194117 2024-04-22T02:53:04.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3213 16 2502 2486 0.00604411 0 - 0.194117 2024-04-22T02:53:05.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3214 16 2502 2486 0.00604223 0 - 0.194117 2024-04-22T02:53:06.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3215 16 2502 2486 0.00604035 0 - 0.194117 2024-04-22T02:53:07.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3216 16 2502 2486 0.00603847 0 - 0.194117 2024-04-22T02:53:08.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3217 16 2502 2486 0.00603659 0 - 0.194117 2024-04-22T02:53:09.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3218 16 2502 2486 0.00603472 0 - 0.194117 2024-04-22T02:53:10.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3219 16 2502 2486 0.00603284 0 - 0.194117 2024-04-22T02:53:11.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:53:11.589190+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:53:11.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:53:11.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3220 16 2502 2486 0.00603097 0 - 0.194117 2024-04-22T02:53:12.586 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3221 16 2502 2486 0.0060291 0 - 0.194117 2024-04-22T02:53:13.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3222 16 2502 2486 0.00602722 0 - 0.194117 2024-04-22T02:53:14.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3223 16 2502 2486 0.00602535 0 - 0.194117 2024-04-22T02:53:15.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3224 16 2502 2486 0.00602349 0 - 0.194117 2024-04-22T02:53:16.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3225 16 2502 2486 0.00602162 0 - 0.194117 2024-04-22T02:53:17.587 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3226 16 2502 2486 0.00601975 0 - 0.194117 2024-04-22T02:53:18.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3227 16 2502 2486 0.00601789 0 - 0.194117 2024-04-22T02:53:19.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3228 16 2502 2486 0.00601602 0 - 0.194117 2024-04-22T02:53:20.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3229 16 2502 2486 0.00601416 0 - 0.194117 2024-04-22T02:53:21.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3230 16 2502 2486 0.0060123 0 - 0.194117 2024-04-22T02:53:22.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3231 16 2502 2486 0.00601044 0 - 0.194117 2024-04-22T02:53:23.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3232 16 2502 2486 0.00600858 0 - 0.194117 2024-04-22T02:53:24.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3233 16 2502 2486 0.00600672 0 - 0.194117 2024-04-22T02:53:25.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3234 16 2502 2486 0.00600486 0 - 0.194117 2024-04-22T02:53:26.588 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3235 16 2502 2486 0.006003 0 - 0.194117 2024-04-22T02:53:27.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3236 16 2502 2486 0.00600115 0 - 0.194117 2024-04-22T02:53:28.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3237 16 2502 2486 0.00599929 0 - 0.194117 2024-04-22T02:53:29.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3238 16 2502 2486 0.00599744 0 - 0.194117 2024-04-22T02:53:30.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3239 16 2502 2486 0.00599559 0 - 0.194117 2024-04-22T02:53:31.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:53:31.591624+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:53:31.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:53:31.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3240 16 2502 2486 0.00599374 0 - 0.194117 2024-04-22T02:53:32.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3241 16 2502 2486 0.00599189 0 - 0.194117 2024-04-22T02:53:33.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3242 16 2502 2486 0.00599004 0 - 0.194117 2024-04-22T02:53:34.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3243 16 2502 2486 0.0059882 0 - 0.194117 2024-04-22T02:53:35.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3244 16 2502 2486 0.00598635 0 - 0.194117 2024-04-22T02:53:36.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3245 16 2502 2486 0.0059845 0 - 0.194117 2024-04-22T02:53:37.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3246 16 2502 2486 0.00598266 0 - 0.194117 2024-04-22T02:53:38.589 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3247 16 2502 2486 0.00598082 0 - 0.194117 2024-04-22T02:53:39.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3248 16 2502 2486 0.00597898 0 - 0.194117 2024-04-22T02:53:40.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3249 16 2502 2486 0.00597714 0 - 0.194117 2024-04-22T02:53:41.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3250 16 2502 2486 0.0059753 0 - 0.194117 2024-04-22T02:53:42.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3251 16 2502 2486 0.00597346 0 - 0.194117 2024-04-22T02:53:43.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3252 16 2502 2486 0.00597162 0 - 0.194117 2024-04-22T02:53:44.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3253 16 2502 2486 0.00596979 0 - 0.194117 2024-04-22T02:53:45.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3254 16 2502 2486 0.00596795 0 - 0.194117 2024-04-22T02:53:46.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3255 16 2502 2486 0.00596612 0 - 0.194117 2024-04-22T02:53:47.590 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3256 16 2502 2486 0.00596429 0 - 0.194117 2024-04-22T02:53:48.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3257 16 2502 2486 0.00596246 0 - 0.194117 2024-04-22T02:53:49.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3258 16 2502 2486 0.00596063 0 - 0.194117 2024-04-22T02:53:50.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3259 16 2502 2486 0.0059588 0 - 0.194117 2024-04-22T02:53:51.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:53:51.593806+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:53:51.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:53:51.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3260 16 2502 2486 0.00595697 0 - 0.194117 2024-04-22T02:53:52.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3261 16 2502 2486 0.00595514 0 - 0.194117 2024-04-22T02:53:53.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3262 16 2502 2486 0.00595332 0 - 0.194117 2024-04-22T02:53:54.591 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3263 16 2502 2486 0.00595149 0 - 0.194117 2024-04-22T02:53:55.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3264 16 2502 2486 0.00594967 0 - 0.194117 2024-04-22T02:53:56.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3265 16 2502 2486 0.00594785 0 - 0.194117 2024-04-22T02:53:57.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3266 16 2502 2486 0.00594602 0 - 0.194117 2024-04-22T02:53:58.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3267 16 2502 2486 0.0059442 0 - 0.194117 2024-04-22T02:53:59.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3268 16 2502 2486 0.00594239 0 - 0.194117 2024-04-22T02:54:00.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3269 16 2502 2486 0.00594057 0 - 0.194117 2024-04-22T02:54:01.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3270 16 2502 2486 0.00593875 0 - 0.194117 2024-04-22T02:54:02.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3271 16 2502 2486 0.00593694 0 - 0.194117 2024-04-22T02:54:03.592 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3272 16 2502 2486 0.00593512 0 - 0.194117 2024-04-22T02:54:04.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3273 16 2502 2486 0.00593331 0 - 0.194117 2024-04-22T02:54:05.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3274 16 2502 2486 0.0059315 0 - 0.194117 2024-04-22T02:54:06.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3275 16 2502 2486 0.00592968 0 - 0.194117 2024-04-22T02:54:07.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3276 16 2502 2486 0.00592787 0 - 0.194117 2024-04-22T02:54:08.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3277 16 2502 2486 0.00592607 0 - 0.194117 2024-04-22T02:54:09.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3278 16 2502 2486 0.00592426 0 - 0.194117 2024-04-22T02:54:10.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3279 16 2502 2486 0.00592245 0 - 0.194117 2024-04-22T02:54:11.593 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:54:11.596279+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:54:11.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:54:11.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3280 16 2502 2486 0.00592065 0 - 0.194117 2024-04-22T02:54:12.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3281 16 2502 2486 0.00591884 0 - 0.194117 2024-04-22T02:54:13.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3282 16 2502 2486 0.00591704 0 - 0.194117 2024-04-22T02:54:14.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3283 16 2502 2486 0.00591523 0 - 0.194117 2024-04-22T02:54:15.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3284 16 2502 2486 0.00591343 0 - 0.194117 2024-04-22T02:54:16.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3285 16 2502 2486 0.00591163 0 - 0.194117 2024-04-22T02:54:17.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3286 16 2502 2486 0.00590983 0 - 0.194117 2024-04-22T02:54:18.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3287 16 2502 2486 0.00590804 0 - 0.194117 2024-04-22T02:54:19.594 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3288 16 2502 2486 0.00590624 0 - 0.194117 2024-04-22T02:54:20.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3289 16 2502 2486 0.00590444 0 - 0.194117 2024-04-22T02:54:21.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3290 16 2502 2486 0.00590265 0 - 0.194117 2024-04-22T02:54:22.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3291 16 2502 2486 0.00590086 0 - 0.194117 2024-04-22T02:54:23.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3292 16 2502 2486 0.00589906 0 - 0.194117 2024-04-22T02:54:24.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3293 16 2502 2486 0.00589727 0 - 0.194117 2024-04-22T02:54:25.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3294 16 2502 2486 0.00589548 0 - 0.194117 2024-04-22T02:54:26.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3295 16 2502 2486 0.00589369 0 - 0.194117 2024-04-22T02:54:27.595 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3296 16 2502 2486 0.0058919 0 - 0.194117 2024-04-22T02:54:28.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3297 16 2502 2486 0.00589012 0 - 0.194117 2024-04-22T02:54:29.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3298 16 2502 2486 0.00588833 0 - 0.194117 2024-04-22T02:54:30.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3299 16 2502 2486 0.00588655 0 - 0.194117 2024-04-22T02:54:31.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:54:31.598802+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:54:31.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:54:31.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3300 16 2502 2486 0.00588476 0 - 0.194117 2024-04-22T02:54:32.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3301 16 2502 2486 0.00588298 0 - 0.194117 2024-04-22T02:54:33.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3302 16 2502 2486 0.0058812 0 - 0.194117 2024-04-22T02:54:34.596 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3303 16 2502 2486 0.00587942 0 - 0.194117 2024-04-22T02:54:35.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3304 16 2502 2486 0.00587764 0 - 0.194117 2024-04-22T02:54:36.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3305 16 2502 2486 0.00587586 0 - 0.194117 2024-04-22T02:54:37.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3306 16 2502 2486 0.00587408 0 - 0.194117 2024-04-22T02:54:38.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3307 16 2502 2486 0.00587231 0 - 0.194117 2024-04-22T02:54:39.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3308 16 2502 2486 0.00587053 0 - 0.194117 2024-04-22T02:54:40.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3309 16 2502 2486 0.00586876 0 - 0.194117 2024-04-22T02:54:41.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3310 16 2502 2486 0.00586698 0 - 0.194117 2024-04-22T02:54:42.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3311 16 2502 2486 0.00586521 0 - 0.194117 2024-04-22T02:54:43.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3312 16 2502 2486 0.00586344 0 - 0.194117 2024-04-22T02:54:44.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3313 16 2502 2486 0.00586167 0 - 0.194117 2024-04-22T02:54:45.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3314 16 2502 2486 0.0058599 0 - 0.194117 2024-04-22T02:54:46.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3315 16 2502 2486 0.00585813 0 - 0.194117 2024-04-22T02:54:47.597 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3316 16 2502 2486 0.00585637 0 - 0.194117 2024-04-22T02:54:48.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3317 16 2502 2486 0.0058546 0 - 0.194117 2024-04-22T02:54:49.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3318 16 2502 2486 0.00585284 0 - 0.194117 2024-04-22T02:54:50.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3319 16 2502 2486 0.00585107 0 - 0.194117 2024-04-22T02:54:51.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:54:51.600570+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:54:51.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:54:51.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3320 16 2502 2486 0.00584931 0 - 0.194117 2024-04-22T02:54:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-22T02:54:52.324 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T02:54:52.359 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T02:54:52.359 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T02:54:52.360 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T02:54:52.360 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:54:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:54:52.366 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-22T02:54:52.388 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T02:54:52.388 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T02:54:52.388 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:54:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:54:52.388 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-22T02:54:52.407 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T02:54:52.407 ERROR:tasks.ceph.osd.2:tried to stop a non-running daemon 2024-04-22T02:54:52.409 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-22T02:54:52.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:54:52.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3321 16 2502 2486 0.00584755 0 - 0.194117 2024-04-22T02:54:53.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3322 16 2502 2486 0.00584579 0 - 0.194117 2024-04-22T02:54:54.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3323 16 2502 2486 0.00584403 0 - 0.194117 2024-04-22T02:54:55.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3324 16 2502 2486 0.00584227 0 - 0.194117 2024-04-22T02:54:56.598 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3325 16 2502 2486 0.00584052 0 - 0.194117 2024-04-22T02:54:57.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3326 16 2502 2486 0.00583876 0 - 0.194117 2024-04-22T02:54:58.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3327 16 2502 2486 0.00583701 0 - 0.194117 2024-04-22T02:54:59.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3328 16 2502 2486 0.00583525 0 - 0.194117 2024-04-22T02:55:00.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3329 16 2502 2486 0.0058335 0 - 0.194117 2024-04-22T02:55:01.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3330 16 2502 2486 0.00583175 0 - 0.194117 2024-04-22T02:55:02.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3331 16 2502 2486 0.00583 0 - 0.194117 2024-04-22T02:55:03.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3332 16 2502 2486 0.00582825 0 - 0.194117 2024-04-22T02:55:04.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3333 16 2502 2486 0.0058265 0 - 0.194117 2024-04-22T02:55:05.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3334 16 2502 2486 0.00582475 0 - 0.194117 2024-04-22T02:55:06.599 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3335 16 2502 2486 0.005823 0 - 0.194117 2024-04-22T02:55:07.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3336 16 2502 2486 0.00582126 0 - 0.194117 2024-04-22T02:55:08.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3337 16 2502 2486 0.00581951 0 - 0.194117 2024-04-22T02:55:09.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3338 16 2502 2486 0.00581777 0 - 0.194117 2024-04-22T02:55:10.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3339 16 2502 2486 0.00581603 0 - 0.194117 2024-04-22T02:55:11.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:55:11.602859+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:55:11.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:55:11.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3340 16 2502 2486 0.00581429 0 - 0.194117 2024-04-22T02:55:12.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3341 16 2502 2486 0.00581255 0 - 0.194117 2024-04-22T02:55:13.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3342 16 2502 2486 0.00581081 0 - 0.194117 2024-04-22T02:55:14.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3343 16 2502 2486 0.00580907 0 - 0.194117 2024-04-22T02:55:15.600 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3344 16 2502 2486 0.00580733 0 - 0.194117 2024-04-22T02:55:16.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3345 16 2502 2486 0.0058056 0 - 0.194117 2024-04-22T02:55:17.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3346 16 2502 2486 0.00580386 0 - 0.194117 2024-04-22T02:55:18.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3347 16 2502 2486 0.00580213 0 - 0.194117 2024-04-22T02:55:19.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3348 16 2502 2486 0.00580039 0 - 0.194117 2024-04-22T02:55:20.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3349 16 2502 2486 0.00579866 0 - 0.194117 2024-04-22T02:55:21.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3350 16 2502 2486 0.00579693 0 - 0.194117 2024-04-22T02:55:22.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3351 16 2502 2486 0.0057952 0 - 0.194117 2024-04-22T02:55:23.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3352 16 2502 2486 0.00579347 0 - 0.194117 2024-04-22T02:55:24.601 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3353 16 2502 2486 0.00579174 0 - 0.194117 2024-04-22T02:55:25.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3354 16 2502 2486 0.00579002 0 - 0.194117 2024-04-22T02:55:26.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3355 16 2502 2486 0.00578829 0 - 0.194117 2024-04-22T02:55:27.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3356 16 2502 2486 0.00578657 0 - 0.194117 2024-04-22T02:55:28.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3357 16 2502 2486 0.00578484 0 - 0.194117 2024-04-22T02:55:29.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3358 16 2502 2486 0.00578312 0 - 0.194117 2024-04-22T02:55:30.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3359 16 2502 2486 0.0057814 0 - 0.194117 2024-04-22T02:55:31.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:55:31.605060+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:55:31.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:55:31.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3360 16 2502 2486 0.00577968 0 - 0.194117 2024-04-22T02:55:32.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3361 16 2502 2486 0.00577796 0 - 0.194117 2024-04-22T02:55:33.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3362 16 2502 2486 0.00577624 0 - 0.194117 2024-04-22T02:55:34.602 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3363 16 2502 2486 0.00577452 0 - 0.194117 2024-04-22T02:55:35.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3364 16 2502 2486 0.00577281 0 - 0.194117 2024-04-22T02:55:36.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3365 16 2502 2486 0.00577109 0 - 0.194117 2024-04-22T02:55:37.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3366 16 2502 2486 0.00576938 0 - 0.194117 2024-04-22T02:55:38.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3367 16 2502 2486 0.00576766 0 - 0.194117 2024-04-22T02:55:39.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3368 16 2502 2486 0.00576595 0 - 0.194117 2024-04-22T02:55:40.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3369 16 2502 2486 0.00576424 0 - 0.194117 2024-04-22T02:55:41.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3370 16 2502 2486 0.00576253 0 - 0.194117 2024-04-22T02:55:42.603 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3371 16 2502 2486 0.00576082 0 - 0.194117 2024-04-22T02:55:43.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3372 16 2502 2486 0.00575911 0 - 0.194117 2024-04-22T02:55:44.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3373 16 2502 2486 0.0057574 0 - 0.194117 2024-04-22T02:55:45.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3374 16 2502 2486 0.0057557 0 - 0.194117 2024-04-22T02:55:46.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3375 16 2502 2486 0.00575399 0 - 0.194117 2024-04-22T02:55:47.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3376 16 2502 2486 0.00575229 0 - 0.194117 2024-04-22T02:55:48.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3377 16 2502 2486 0.00575058 0 - 0.194117 2024-04-22T02:55:49.604 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3378 16 2502 2486 0.00574888 0 - 0.194117 2024-04-22T02:55:50.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3379 16 2502 2486 0.00574718 0 - 0.194117 2024-04-22T02:55:51.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:55:51.607201+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:55:51.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:55:51.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3380 16 2502 2486 0.00574548 0 - 0.194117 2024-04-22T02:55:52.511 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None, False)> 2024-04-22T02:55:52.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3381 16 2502 2486 0.00574378 0 - 0.194117 2024-04-22T02:55:53.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3382 16 2502 2486 0.00574208 0 - 0.194117 2024-04-22T02:55:54.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3383 16 2502 2486 0.00574038 0 - 0.194117 2024-04-22T02:55:55.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3384 16 2502 2486 0.00573869 0 - 0.194117 2024-04-22T02:55:56.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3385 16 2502 2486 0.00573699 0 - 0.194117 2024-04-22T02:55:57.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3386 16 2502 2486 0.0057353 0 - 0.194117 2024-04-22T02:55:58.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3387 16 2502 2486 0.0057336 0 - 0.194117 2024-04-22T02:55:59.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3388 16 2502 2486 0.00573191 0 - 0.194117 2024-04-22T02:56:00.605 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3389 16 2502 2486 0.00573022 0 - 0.194117 2024-04-22T02:56:01.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3390 16 2502 2486 0.00572853 0 - 0.194117 2024-04-22T02:56:02.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3391 16 2502 2486 0.00572684 0 - 0.194117 2024-04-22T02:56:03.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3392 16 2502 2486 0.00572515 0 - 0.194117 2024-04-22T02:56:04.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3393 16 2502 2486 0.00572347 0 - 0.194117 2024-04-22T02:56:05.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3394 16 2502 2486 0.00572178 0 - 0.194117 2024-04-22T02:56:06.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3395 16 2502 2486 0.00572009 0 - 0.194117 2024-04-22T02:56:07.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3396 16 2502 2486 0.00571841 0 - 0.194117 2024-04-22T02:56:08.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3397 16 2502 2486 0.00571673 0 - 0.194117 2024-04-22T02:56:09.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3398 16 2502 2486 0.00571504 0 - 0.194117 2024-04-22T02:56:10.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3399 16 2502 2486 0.00571336 0 - 0.194117 2024-04-22T02:56:11.606 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:56:11.609278+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:56:11.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:56:11.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3400 16 2502 2486 0.00571168 0 - 0.194117 2024-04-22T02:56:12.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3401 16 2502 2486 0.00571 0 - 0.194117 2024-04-22T02:56:13.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3402 16 2502 2486 0.00570832 0 - 0.194117 2024-04-22T02:56:14.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3403 16 2502 2486 0.00570665 0 - 0.194117 2024-04-22T02:56:15.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3404 16 2502 2486 0.00570497 0 - 0.194117 2024-04-22T02:56:16.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3405 16 2502 2486 0.00570329 0 - 0.194117 2024-04-22T02:56:17.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3406 16 2502 2486 0.00570162 0 - 0.194117 2024-04-22T02:56:18.607 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3407 16 2502 2486 0.00569995 0 - 0.194117 2024-04-22T02:56:19.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3408 16 2502 2486 0.00569827 0 - 0.194117 2024-04-22T02:56:20.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3409 16 2502 2486 0.0056966 0 - 0.194117 2024-04-22T02:56:21.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3410 16 2502 2486 0.00569493 0 - 0.194117 2024-04-22T02:56:22.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3411 16 2502 2486 0.00569326 0 - 0.194117 2024-04-22T02:56:23.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3412 16 2502 2486 0.00569159 0 - 0.194117 2024-04-22T02:56:24.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3413 16 2502 2486 0.00568993 0 - 0.194117 2024-04-22T02:56:25.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3414 16 2502 2486 0.00568826 0 - 0.194117 2024-04-22T02:56:26.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3415 16 2502 2486 0.00568659 0 - 0.194117 2024-04-22T02:56:27.608 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3416 16 2502 2486 0.00568493 0 - 0.194117 2024-04-22T02:56:28.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3417 16 2502 2486 0.00568327 0 - 0.194117 2024-04-22T02:56:29.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3418 16 2502 2486 0.0056816 0 - 0.194117 2024-04-22T02:56:30.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3419 16 2502 2486 0.00567994 0 - 0.194117 2024-04-22T02:56:31.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:56:31.611692+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:56:31.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:56:31.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3420 16 2502 2486 0.00567828 0 - 0.194117 2024-04-22T02:56:32.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3421 16 2502 2486 0.00567662 0 - 0.194117 2024-04-22T02:56:33.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3422 16 2502 2486 0.00567496 0 - 0.194117 2024-04-22T02:56:34.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3423 16 2502 2486 0.0056733 0 - 0.194117 2024-04-22T02:56:35.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3424 16 2502 2486 0.00567165 0 - 0.194117 2024-04-22T02:56:36.609 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3425 16 2502 2486 0.00566999 0 - 0.194117 2024-04-22T02:56:37.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3426 16 2502 2486 0.00566834 0 - 0.194117 2024-04-22T02:56:38.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3427 16 2502 2486 0.00566668 0 - 0.194117 2024-04-22T02:56:39.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3428 16 2502 2486 0.00566503 0 - 0.194117 2024-04-22T02:56:40.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3429 16 2502 2486 0.00566338 0 - 0.194117 2024-04-22T02:56:41.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3430 16 2502 2486 0.00566172 0 - 0.194117 2024-04-22T02:56:42.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3431 16 2502 2486 0.00566007 0 - 0.194117 2024-04-22T02:56:43.610 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3432 16 2502 2486 0.00565843 0 - 0.194117 2024-04-22T02:56:44.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3433 16 2502 2486 0.00565678 0 - 0.194117 2024-04-22T02:56:45.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3434 16 2502 2486 0.00565513 0 - 0.194117 2024-04-22T02:56:46.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3435 16 2502 2486 0.00565348 0 - 0.194117 2024-04-22T02:56:47.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3436 16 2502 2486 0.00565184 0 - 0.194117 2024-04-22T02:56:48.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3437 16 2502 2486 0.00565019 0 - 0.194117 2024-04-22T02:56:49.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3438 16 2502 2486 0.00564855 0 - 0.194117 2024-04-22T02:56:50.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3439 16 2502 2486 0.00564691 0 - 0.194117 2024-04-22T02:56:51.611 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-04-22T02:56:51.614051+0000 min lat: 0.0137586 max lat: 15.1997 avg lat: 0.194117 2024-04-22T02:56:51.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-22T02:56:51.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3440 16 2502 2486 0.00564527 0 - 0.194117 2024-04-22T02:56:52.512 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None, False)> 2024-04-22T02:56:52.513 INFO:tasks.ceph.osd.3:Stopped 2024-04-22T02:56:52.513 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T02:56:52.513 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T02:56:52.513 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:56:52.514 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T02:56:52.514 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T02:56:52.514 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T02:56:52.514 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T02:56:52.515 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T02:56:52.515 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T02:56:52.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-22T02:56:52.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T02:56:52.572 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T02:56:52.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3441 16 2502 2486 0.00564363 0 - 0.194117 2024-04-22T02:56:52.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T02:56:52.646 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T02:56:52.735 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T02:56:52.736 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T02:56:52.814 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T02:56:52.815 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-22T02:56:52.871 INFO:tasks.ceph:Archiving mon data... 2024-04-22T02:56:52.873 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/data/mon.a.tgz 2024-04-22T02:56:52.873 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-04-22T02:56:52.889 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.PjBpwHzOzL 2024-04-22T02:56:52.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.PjBpwHzOzL 2024-04-22T02:56:53.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.PjBpwHzOzL 2024-04-22T02:56:53.105 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.PjBpwHzOzL is 320KB 2024-04-22T02:56:53.201 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.PjBpwHzOzL 2024-04-22T02:56:53.216 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T02:56:53.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T02:56:53.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T02:56:53.335 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T02:56:53.352 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T02:56:53.352 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T02:56:53.354 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi107/crash 2024-04-22T02:56:53.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T02:56:53.398 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi150/crash 2024-04-22T02:56:53.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T02:56:53.470 INFO:tasks.ceph:Compressing logs... 2024-04-22T02:56:53.471 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T02:56:53.473 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T02:56:53.507 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27057.log 2024-04-22T02:56:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T02:56:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27177.log 2024-04-22T02:56:53.508 INFO:teuthology.orchestra.run.smithi150.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27057.log.gz 2024-04-22T02:56:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-22T02:56:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27177.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T02:56:53.509 INFO:teuthology.orchestra.run.smithi150.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27177.log.gz 2024-04-22T02:56:53.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27097.log 2024-04-22T02:56:53.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27137.log 2024-04-22T02:56:53.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27097.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27097.log.gz 2024-04-22T02:56:53.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27137.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27137.log.gz 2024-04-22T02:56:53.557 INFO:teuthology.orchestra.run.smithi150.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32499.log 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28109.log 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34103.log 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.32499.log.gz 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31670.log 2024-04-22T02:56:53.634 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28109.log.gz 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35556.log 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34103.log.gz 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31954.log 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31670.log.gz 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32478.log 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35556.log.gz 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37178.log 2024-04-22T02:56:53.635 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31954.log.gz 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30636.log 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32478.log.gz 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28457.log 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.37178.log.gz 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29897.log 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30636.log.gz 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30378.log 2024-04-22T02:56:53.636 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28457.log.gz 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32607.log 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29897.log.gz 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35383.log 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30378.log.gz 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28218.log 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32607.log.gz 2024-04-22T02:56:53.637 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34744.log 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35383.log.gz 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37515.log 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28218.log.gz 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36741.log 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34744.log.gz 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36568.log 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37515.log.gz 2024-04-22T02:56:53.638 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36741.log.gz 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37639.log 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30498.log 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37546.log 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz 2024-04-22T02:56:53.639 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37057.log 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30498.log.gz 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.37546.log.gz 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29464.log 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37057.log.gz 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27776.log 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35858.log 2024-04-22T02:56:53.640 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29464.log.gz 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29562.log 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27776.log.gz 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29846.log 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35858.log.gz 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29562.log.gz 2024-04-22T02:56:53.641 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31281.log 2024-04-22T02:56:53.642 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29846.log.gz 2024-04-22T02:56:53.642 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35776.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T02:56:53.642 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-04-22T02:56:53.642 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32627.log 2024-04-22T02:56:53.642 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31281.log.gz 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30198.log 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28880.log 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32627.log.gz 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28008.log 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30198.log.gz 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32729.log 2024-04-22T02:56:53.643 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28880.log.gz 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31419.log 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28008.log.gz 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr: 88.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32429.log 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32729.log.gz 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34150.log 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31419.log.gz 2024-04-22T02:56:53.644 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33943.log 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32429.log.gz 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27832.log 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34150.log.gz 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29029.log 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33943.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33943.log.gz 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32846.log 2024-04-22T02:56:53.645 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27832.log.gz 2024-04-22T02:56:53.646 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30736.log 2024-04-22T02:56:53.646 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29029.log.gz 2024-04-22T02:56:53.646 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.0.32846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31827.log 2024-04-22T02:56:53.646 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32005.log 2024-04-22T02:56:53.646 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30736.log.gz 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33849.log 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31827.log.gz 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32005.log.gz 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31201.log 2024-04-22T02:56:53.647 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33849.log.gz 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33343.log 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.31201.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31201.log.gz 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30458.log 2024-04-22T02:56:53.648 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33343.log.gz 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30854.log 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30458.log.gz 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35645.log 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30854.log.gz 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr: 87.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T02:56:53.649 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37392.log 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34197.log 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35645.log.gz 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37809.log 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37392.log.gz 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30716.log 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34197.log.gz 2024-04-22T02:56:53.650 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36610.log 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.37809.log.gz 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37239.log 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30716.log.gz 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31321.log 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36610.log.gz 2024-04-22T02:56:53.651 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34023.log 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37239.log.gz 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34063.log 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31321.log.gz 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34954.log 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34023.log.gz 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29052.log 2024-04-22T02:56:53.652 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34063.log.gz 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37779.log 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34954.log.gz 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.0.29052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30547.log 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33509.log 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37779.log.gz 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35980.log 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30547.log.gz 2024-04-22T02:56:53.653 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29764.log 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33509.log.gz 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.0.35980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29764.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29764.log.gz 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29212.log 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28454.log 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2024-04-22T02:56:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37148.log 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28277.log 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28454.log.gz 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36652.log 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37148.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37148.log.gz 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28277.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.28692.log 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28277.log.gz 2024-04-22T02:56:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30072.log 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36652.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36652.log.gz 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28692.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28692.log.gz 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36437.log 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30072.log.gz 2024-04-22T02:56:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29163.log 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.36437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30596.log 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36437.log.gz 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27924.log 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29163.log.gz 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33112.log 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30596.log.gz 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28107.log 2024-04-22T02:56:53.657 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27924.log.gz 2024-04-22T02:56:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33112.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T02:56:53.658 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33112.log.gz 2024-04-22T02:56:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35122.log 2024-04-22T02:56:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28107.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28107.log.gz 2024-04-22T02:56:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37870.log 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35122.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35122.log.gz 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.37455.log 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37870.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.37870.log.gz 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37455.log: gzip -5 --verbose 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37455.log.gz 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.33383.log 2024-04-22T02:56:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30676.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28397.log 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33383.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33383.log.gz 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.32089.log 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28397.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28397.log.gz 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36834.log 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32089.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32089.log.gz 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.37017.log 2024-04-22T02:56:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36834.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36834.log.gz 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35206.log 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37017.log: gzip -5 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37017.log.gz 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35729.log 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35206.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35206.log.gz 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30903.log 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35729.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35729.log.gz 2024-04-22T02:56:53.661 INFO:teuthology.orchestra.run.smithi107.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33161.log 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30903.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30903.log.gz 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28958.log 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32047.log 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33161.log: /var/log/ceph/ceph-client.admin.28958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30296.log 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33161.log.gz 2024-04-22T02:56:53.662 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28958.log.gz 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32047.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32047.log.gz 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28110.log 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29806.log 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30296.log.gz 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28110.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28110.log.gz 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33255.log 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29806.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29806.log.gz 2024-04-22T02:56:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33255.log: gzip -5 --verbose 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33255.log.gz 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.30114.log 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37302.log 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31112.log 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30114.log.gz 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37840.log 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31112.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37302.log.gz 2024-04-22T02:56:53.664 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31112.log.gz 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37669.log 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31370.log 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37840.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37840.log.gz 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37669.log: 8.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32380.log 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37669.log.gz 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32903.log 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32380.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31370.log.gz 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32380.log.gz 2024-04-22T02:56:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35038.log 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36388.log 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32903.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32903.log.gz 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35038.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35038.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34534.log 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37271.log 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34534.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36388.log.gz 2024-04-22T02:56:53.666 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34534.log.gz 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30338.log 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30030.log 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37271.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37271.log.gz 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30338.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30338.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29980.log 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30030.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30030.log.gz 2024-04-22T02:56:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34618.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28608.log 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gz 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37425.log 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28608.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29980.log.gz 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28608.log.gz 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30247.log 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34828.log 2024-04-22T02:56:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37425.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37425.log.gz 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30247.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30247.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36694.log 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34828.log: /var/log/ceph/ceph-client.admin.34870.log: 50.4% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34828.log.gz 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35687.log 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36973.log 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36694.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36694.log.gz 2024-04-22T02:56:53.669 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35687.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.35687.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33782.log 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30943.log 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36973.log: /var/log/ceph/ceph-client.admin.33782.log: 49.6% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36973.log.gz 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33782.log.gz 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37485.log 2024-04-22T02:56:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32311.log 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30943.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30943.log.gz 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37485.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37485.log.gz 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35818.log 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35910.log 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32311.log: /var/log/ceph/ceph-client.admin.35818.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32311.log.gz 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35818.log.gz 2024-04-22T02:56:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33423.log 2024-04-22T02:56:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34702.log 2024-04-22T02:56:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35910.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35910.log.gz 2024-04-22T02:56:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33423.log: 49.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35603.log 2024-04-22T02:56:53.672 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33423.log.gz 2024-04-22T02:56:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-04-22T02:56:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35603.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34702.log.gz 2024-04-22T02:56:53.673 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35603.log.gz 2024-04-22T02:56:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37609.log 2024-04-22T02:56:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36874.log 2024-04-22T02:56:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33896.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-04-22T02:56:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37609.log: 8.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31241.log 2024-04-22T02:56:53.674 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37609.log.gz 2024-04-22T02:56:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37903.log 2024-04-22T02:56:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36874.log: /var/log/ceph/ceph-client.admin.31241.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36874.log.gz 2024-04-22T02:56:53.674 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31241.log.gz 2024-04-22T02:56:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34912.log 2024-04-22T02:56:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-04-22T02:56:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37903.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37903.log.gz 2024-04-22T02:56:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34912.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34912.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33983.log 2024-04-22T02:56:53.675 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28453.log 2024-04-22T02:56:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37088.log: /var/log/ceph/ceph-client.admin.33983.log: 49.6% 9.6% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-04-22T02:56:53.677 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33983.log.gz 2024-04-22T02:56:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-04-22T02:56:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35425.log 2024-04-22T02:56:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28453.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28453.log.gz 2024-04-22T02:56:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28959.log 2024-04-22T02:56:53.678 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-04-22T02:56:53.679 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28108.log 2024-04-22T02:56:53.679 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28959.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35425.log.gz 2024-04-22T02:56:53.679 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28959.log.gz 2024-04-22T02:56:53.679 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37934.log 2024-04-22T02:56:53.679 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37208.log 2024-04-22T02:56:53.680 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28108.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28108.log.gz 2024-04-22T02:56:53.680 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37934.log: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.37934.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28276.log 2024-04-22T02:56:53.680 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.680 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31468.log 2024-04-22T02:56:53.680 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37208.log: /var/log/ceph/ceph-client.admin.28276.log: 50.0% 8.0% -- replaced with /var/log/ceph/ceph-client.admin.37208.log.gz 2024-04-22T02:56:53.681 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28276.log.gz 2024-04-22T02:56:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34410.log 2024-04-22T02:56:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36914.log 2024-04-22T02:56:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31468.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31468.log.gz 2024-04-22T02:56:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34410.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34410.log.gz 2024-04-22T02:56:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37699.log 2024-04-22T02:56:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-04-22T02:56:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37699.log: /var/log/ceph/ceph-client.admin.36914.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-04-22T02:56:53.682 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37699.log.gz 2024-04-22T02:56:53.682 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34338.log 2024-04-22T02:56:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31739.log 2024-04-22T02:56:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35467.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-04-22T02:56:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34338.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34338.log.gz 2024-04-22T02:56:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31780.log 2024-04-22T02:56:53.683 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33297.log 2024-04-22T02:56:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31739.log: /var/log/ceph/ceph-client.admin.31780.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31739.log.gz 2024-04-22T02:56:53.684 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31780.log.gz 2024-04-22T02:56:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37362.log 2024-04-22T02:56:53.684 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29394.log 2024-04-22T02:56:53.684 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33297.log.gz 2024-04-22T02:56:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-04-22T02:56:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29394.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37362.log.gz 2024-04-22T02:56:53.685 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29394.log.gz 2024-04-22T02:56:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31872.log 2024-04-22T02:56:53.685 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33210.log 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34660.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31872.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31872.log.gz 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34291.log 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27525.log 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34291.log: /var/log/ceph/ceph-client.admin.33210.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33210.log.gz 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34291.log.gz 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27882.log 2024-04-22T02:56:53.686 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27525.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27525.log.gz 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27882.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27882.log.gz 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33599.log 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33032.log 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33599.log: /var/log/ceph/ceph-client.admin.36037.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33599.log.gz 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31914.log 2024-04-22T02:56:53.687 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33698.log 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33032.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33032.log.gz 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31914.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31914.log.gz 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29513.log 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27289.log 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29513.log: /var/log/ceph/ceph-client.admin.33698.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33698.log.gz 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29513.log.gz 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32794.log 2024-04-22T02:56:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35952.log 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27289.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27289.log.gz 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32794.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32794.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34244.log 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35952.log: /var/log/ceph/ceph-client.admin.34576.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35952.log.gz 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33463.log 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32567.log 2024-04-22T02:56:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34244.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34244.log.gz 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33463.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33463.log.gz 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27368.log 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37579.log: /var/log/ceph/ceph-client.admin.32567.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32567.log.gz 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2024-04-22T02:56:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27368.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27368.log.gz 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.33072.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29114.log 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35164.log 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29114.log: /var/log/ceph/ceph-client.admin.32952.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29114.log.gz 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32992.log 2024-04-22T02:56:53.691 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28278.log 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35164.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35164.log.gz 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32992.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32992.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28734.log 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30805.log 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28278.log: /var/log/ceph/ceph-client.admin.28734.log: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28278.log.gz 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28734.log.gz 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34996.log 2024-04-22T02:56:53.692 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37332.log 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30805.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30805.log.gz 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.34996.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37118.log 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34996.log.gz 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36259.log 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.37118.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.37332.log.gz 2024-04-22T02:56:53.693 INFO:teuthology.orchestra.run.smithi107.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.37118.log.gz 2024-04-22T02:56:53.694 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30156.log 2024-04-22T02:56:53.694 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31023.log 2024-04-22T02:56:53.694 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36259.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36259.log.gz 2024-04-22T02:56:53.695 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30156.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30156.log.gz 2024-04-22T02:56:53.695 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28650.log 2024-04-22T02:56:53.695 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29939.log 2024-04-22T02:56:53.695 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28650.log: /var/log/ceph/ceph-client.admin.31023.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31023.log.gz 2024-04-22T02:56:53.696 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28650.log.gz 2024-04-22T02:56:53.696 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26221.log 2024-04-22T02:56:53.696 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36520.log 2024-04-22T02:56:53.696 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29939.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29939.log.gz 2024-04-22T02:56:53.697 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph.tmp-client.admin.26221.log: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26221.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36789.log 2024-04-22T02:56:53.697 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.697 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2024-04-22T02:56:53.697 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36520.log: /var/log/ceph/ceph-client.admin.36789.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36789.log.gz 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36520.log.gz 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30418.log 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.35501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32667.log 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27553.log 2024-04-22T02:56:53.698 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.32667.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30418.log.gz 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32667.log.gz 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31063.log 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28565.log 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27553.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27553.log.gz 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31063.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31063.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35336.log 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36479.log 2024-04-22T02:56:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28565.log: /var/log/ceph/ceph-client.admin.35336.log: 50.0% 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28565.log.gz 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35336.log.gz 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28776.log 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32773.log 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36479.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36479.log.gz 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28776.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28776.log.gz 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27966.log 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36348.log 2024-04-22T02:56:53.700 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27966.log: /var/log/ceph/ceph-client.admin.32773.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32773.log.gz 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27966.log.gz 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28818.log 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36306.log 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36348.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36348.log.gz 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28818.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28818.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34786.log 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:53.701 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31161.log 2024-04-22T02:56:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.36306.log: /var/log/ceph/ceph-client.admin.34786.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36306.log.gz 2024-04-22T02:56:53.702 INFO:teuthology.orchestra.run.smithi107.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34786.log.gz 2024-04-22T02:56:53.702 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31161.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31161.log.gz 2024-04-22T02:56:53.703 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3442 16 2502 2486 0.00564199 0 - 0.194117 2024-04-22T02:56:53.704 INFO:teuthology.orchestra.run.smithi107.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.35980.log.gz 2024-04-22T02:56:53.704 INFO:teuthology.orchestra.run.smithi107.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.32846.log.gz 2024-04-22T02:56:53.704 INFO:teuthology.orchestra.run.smithi107.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.29052.log.gz 2024-04-22T02:56:53.762 INFO:teuthology.orchestra.run.smithi107.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-22T02:56:54.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3443 16 2502 2486 0.00564035 0 - 0.194117 2024-04-22T02:56:55.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3444 16 2502 2486 0.00563871 0 - 0.194117 2024-04-22T02:56:55.710 INFO:teuthology.orchestra.run.smithi150.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T02:56:55.886 INFO:teuthology.orchestra.run.smithi107.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T02:56:56.156 INFO:teuthology.orchestra.run.smithi150.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-22T02:56:56.157 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-22T02:56:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m2.667s 2024-04-22T02:56:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m4.770s 2024-04-22T02:56:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.168s 2024-04-22T02:56:56.563 INFO:teuthology.orchestra.run.smithi107.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T02:56:56.565 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-04-22T02:56:56.565 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m3.081s 2024-04-22T02:56:56.565 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m5.974s 2024-04-22T02:56:56.565 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.309s 2024-04-22T02:56:56.566 INFO:tasks.ceph:Archiving logs... 2024-04-22T02:56:56.566 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi107/log 2024-04-22T02:56:56.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T02:56:56.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3445 16 2502 2486 0.00563707 0 - 0.194117 2024-04-22T02:56:57.626 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3446 16 2502 2486 0.00563544 0 - 0.194117 2024-04-22T02:56:57.820 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi150/log 2024-04-22T02:56:57.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T02:56:58.624 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3447 16 2502 2486 0.0056338 0 - 0.194117 2024-04-22T02:56:58.710 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-22T02:56:58.732 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T02:56:58.780 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2856, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2846, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2609, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 2601, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-22T02:56:58.797 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T02:56:58.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T02:56:58.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T02:56:58.890 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T02:56:58.899 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T02:56:58.899 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T02:56:58.901 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T02:56:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-04-22T02:56:58.918 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -532us[ -532us] +/- 68ms 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 7 377 113 -384us[ -408us] +/- 67ms 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +990us[ +990us] +/- 73ms 2024-04-22T02:56:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T02:56:58.922 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T02:56:58.923 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-04-22T02:56:58.923 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 177 -507us[ -510us] +/- 66ms 2024-04-22T02:56:58.923 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv02.front.sepia.ceph.com 3 7 377 49 -442us[ -446us] +/- 67ms 2024-04-22T02:56:58.923 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +957us[ +957us] +/- 73ms 2024-04-22T02:56:58.923 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T02:56:58.924 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T02:56:58.933 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T02:56:58.933 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T02:56:58.942 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T02:56:59.029 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T02:56:59.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T02:56:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:56:59.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T02:56:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:56:59.153 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T02:56:59.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T02:56:59.179 INFO:teuthology.task.internal:Duration was 4144.444556 seconds 2024-04-22T02:56:59.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T02:56:59.188 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T02:56:59.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T02:56:59.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T02:56:59.227 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T02:56:59.232 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T02:56:59.469 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T02:56:59.469 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-04-22T02:56:59.470 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T02:56:59.493 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-04-22T02:56:59.493 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T02:56:59.535 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T02:56:59.536 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T02:56:59.538 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T02:56:59.601 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T02:56:59.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T02:56:59.605 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T02:56:59.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3448 16 2502 2486 0.00563217 0 - 0.194117 2024-04-22T02:56:59.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T02:56:59.757 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T02:56:59.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T02:56:59.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T02:56:59.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T02:56:59.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T02:56:59.824 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T02:56:59.843 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-04-22T02:56:59.866 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-04-22T02:56:59.891 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T02:56:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:56:59.918 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T02:56:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T02:56:59.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T02:56:59.955 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T02:56:59.956 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi107 2024-04-22T02:56:59.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T02:56:59.990 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104/remote/smithi150 2024-04-22T02:56:59.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T02:57:00.024 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T02:57:00.025 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T02:57:00.027 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T02:57:00.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T02:57:00.087 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T02:57:00.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T02:57:00.098 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T02:57:00.098 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T02:57:00.101 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T02:57:00.112 INFO:teuthology.orchestra.run.smithi107.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 02:57 /home/ubuntu/cephtest 2024-04-22T02:57:00.128 INFO:teuthology.orchestra.run.smithi150.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 02:57 /home/ubuntu/cephtest 2024-04-22T02:57:00.129 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T02:57:00.139 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T02:57:00.150 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi107 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-04-22T02:57:00.151 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T02:57:00.175 DEBUG:teuthology.nuke:shortname: smithi107 2024-04-22T02:57:00.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T02:57:00.198 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:41:50.462463', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKktq2SK+j+hRsjA8azgEg0Ch7sfdBKBzBmFRSDqNg+A+9nIkMS2Pj/FeT2gIg7W6d0c/VDs8uku7iiOfbovN4='} 2024-04-22T02:57:00.201 DEBUG:teuthology.nuke:shortname: smithi150 2024-04-22T02:57:00.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T02:57:00.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:41:50.460946', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6fG8tz02/G/GPRaj1/U74lFo8yZzXxkcBYrMbQh8/NNLzWU098PBo/Lux++g9b6vl3k1FG/SmkSl2fQFdkDrY='} 2024-04-22T02:57:00.256 INFO:teuthology.orchestra.console.smithi107:Power off 2024-04-22T02:57:00.257 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T02:57:00.282 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-04-22T02:57:00.282 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:00.300 INFO:teuthology.orchestra.console.smithi150:Power off 2024-04-22T02:57:00.300 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T02:57:00.311 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-22T02:57:00.325 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-04-22T02:57:00.325 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:00.350 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-04-22T02:57:00.613 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3449 16 2502 2486 0.00563054 0 - 0.194117 2024-04-22T02:57:01.612 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3450 16 2502 2486 0.0056289 0 - 0.194117 2024-04-22T02:57:02.613 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3451 16 2502 2486 0.00562727 0 - 0.194117 2024-04-22T02:57:04.313 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:04.352 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:04.427 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-22T02:57:04.462 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-04-22T02:57:08.430 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:08.463 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T02:57:08.540 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-22T02:57:08.574 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-04-22T02:57:08.642 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-04-22T02:57:08.676 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-04-22T02:57:08.781 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} duration: 4144.444555521011 failure_reason: reached maximum tries (501) after waiting for 3000 seconds flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2bcda724b5df4ad394b7676fe8d228e9 status: fail success: false 2024-04-22T02:57:08.781 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T02:57:08.885 INFO:teuthology.run:FAIL